Transcript
®
Tiered Adaptive Storage™ Hardware Guide
HR85-8500-A
Cray Private
© 2014 Cray Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE: The Computer Software is delivered as “Commercial Computer Software” as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. The following are trademarks of Cray Inc. and are registered in the United States and other countries: CRAY and design, SONEXION, URIKA, and YARCDATA. The following are trademarks of Cray Inc.: ACE, APPRENTICE2, CHAPEL, CLUSTER CONNECT, CRAYPAT, CRAYPORT, ECOPHLEX, LIBSCI, NODEKARE, THREADSTORM. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners. Direct requests for copies of publications to: Mail:
Cray Inc. Logistics PO Box 6000 Chippewa Falls, WI 54729-0080 USA
E-mail: Fax:
[email protected] +1 715 726 4602
Direct comments about this publication to: Mail:
Cray Inc.. Customer Documentation and Training P.O. Box 6000 Chippewa Falls, WI 54729-0080 USA
E-mail: Fax:
[email protected] +1 715 726 4991
Record of Revision Publication Number
Description
HR85-8500-A
April 2014 Revision 0 June 2014 Revision A: Added section on TAS Connector.
HR85-8500-A
HR85-8500-A
Cray Private
3
Tiered Adaptive Storage™ Hardware Guide
4
Cray Private
HR85-8500-A
Contents
Contents Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 About this Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Definition of a Cray TAS System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Product Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Additional Information Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Safety Precautions and Hazards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Chapter 2: System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Hardware Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 TAS Gateway System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 TAS Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Full TAS Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Full TAS Basic Configuration Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 MDC PCIe Slot Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 DM PCIe Slot Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Metadata Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Basic Tape Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 FC Full TAS Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 IB Full TAS Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 Hardware Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Cray Integrated Management Server (CIMS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Metadata Controller (MDC) and Data Mover (DM) Nodes . . . . . . . . . . . . . . . . . . . . . . .28 Administrative Block Storage (ABS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Data Cache Block Storage (DCBS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 Data Archive Storage Tape (DAST) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 Chapter 3: Site Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Site Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 System Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Shipping Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Site Access Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 Computer Room Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Electrical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 Equipment Grounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 Floor Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 Chapter 4: Hardware Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Rack and PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 TAS System Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Power Distribution Units (PDUs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Cray Integrated Management Server (CIMS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 CIMS Controls and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 CIMS IO Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
HR85-8500-A
Cray Private
1
Tiered Adaptive Storage™ Hardware Guide
Metadata Controller (MDC) and Data Mover (DM) Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 MDC and DM Controls and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 MDC and DM IO Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 CIMS, MDC, and DM Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Keyboard, Video, and Mouse (KVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Hotkey Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 TAS Metadata Block Storage Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Controller Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Administrative Block Storage (ABS) SAS Interface, Controls, and Indicators . . . . 54 Data Cache Block Storage (DCBS) FC Interface, Controls, and Indicators . . . . . . . 63 Data Cache Block (DCBS) IB Interface, Controls, and Indicators . . . . . . . . . . . . . . 70 ABS and DCBS LED Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Drive Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 ABS Drive Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 DCBS (FC) Drive Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 DCBS (IB) Drive Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Disk Drive Trays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 ABS 24-bay Drive Tray Controls and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 DCBS 60-bay Drive Tray Controls and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Tape Archive Storage Tier (TAST) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Data Network Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 InfiniBand Data Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Switches and Cables (Mellanox SX6036) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Switch Platform Hardware Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Switch Status LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Power Supply Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 UID LED Switch Identifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Port Connector LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Fibre Channel Data Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Switches and Cables (QLogic 5802V) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Chassis Controls and LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Heartbeat LED Blink Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 FC Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Logged-In LED Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Ethernet Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Serial Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Power Supplies and Fans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Ethernet Management Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Switches and Cables (Dell PowerConnect 5548) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Dell PowerConnect 5548 Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Buttons and LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Gigabit Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 HDMI Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 SFP LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Chapter 5: Block Storage Tray ID Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 ABS (NetApp 2700) and DCBS (NetApp 5500) Tray ID Indicator Codes . . . . . . . . . . . . . . 113
2
Cray Private
HR85-8500-A
Contents
Tray ID Failure Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113 Controller-Drive Tray Sequence Code Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114 Lock-Down Codes for the Controller-Drive Tray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114 Tray ID Indicator Display Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116 Tray ID Indicator Display for the Drive Trays . . . . . . . . . . . . . . . . . . . . . . . . . . . .117 DCBS (NetApp 5400) Tray ID Indicator Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118 Sequence Code Definitions for the Controller-Drive Trays . . . . . . . . . . . . . . . . . . .118 Lock-Down Codes for the Controller-Drive Trays . . . . . . . . . . . . . . . . . . . . . . . . .119 Diagnostic Code Sequences for the Controller-Drive Trays . . . . . . . . . . . . . . . . . .120 Tray ID Indicator Display for the ESMs on the Drive Trays . . . . . . . . . . . . . . . . . .121
HR85-8500-A
Cray Private
3
Tiered Adaptive Storage™ Hardware Guide
4
Cray Private
HR85-8500-A
Contents
List of Figures Figure 2-1: TAS Gateway Minimum Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . .8 Figure 2-2: TAS Gateway (Front View) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Figure 2-3: ETHSW1, TAS Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Figure 2-4: CIMS, TAS Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Figure 2-5: Secondary MDC 2, TAS Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Figure 2-6: Primary MDC 1, TAS Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Figure 2-7: Administrative Block Storage (ABS), TAS Gateway . . . . . . . . . . . . . . . . . . . . . . .9 Figure 2-8: TAS Connector Manager Node with IB Lustre . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Figure 2-9: TAS Connector Agent on a TAS DM with IB Lustre and IB TAS . . . . . . . . . . . .10 Figure 2-10: TAS Connector Manager SAS Connections, Secondary TCM2 . . . . . . . . . . . . .10 Figure 2-11: TAS Connector Manager SAS Connections, Primary TCM 1 . . . . . . . . . . . . . .11 Figure 2-12: TAS Connector Manager SAS Connections, Administrative Block Storage . . .11 Figure 2-13: Full TAS System (Front View) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Figure 2-14: Full TAS System (Rear View) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 Figure 2-15: MDC with FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Figure 2-16: MDC with IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Figure 2-17: DM with FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Figure 2-18: DM with IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 Figure 2-19: Secondary MDC 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Figure 2-20: Primary MDC 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Figure 2-21: Administrative Block Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Figure 2-22: FCSW2, Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Figure 2-23: FCSW1, Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Figure 2-24: Secondary MDC 2, FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Figure 2-25: Primary MDC 1, FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Figure 2-26: Secondary DM 2, FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Figure 2-27: Primary DM 1, FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Figure 2-28: DCBS, FC Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Figure 2-29: IBSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-30: IBSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-31: Secondary MDC 2, IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-32: Primary MDC 1, IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-33: Secondary DM 2, IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-34: Primary DM 1, IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Figure 2-35: Primary DM 1, IB Data Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Figure 2-36: FCSW2, Tape Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Figure 2-37: FCSW1, Tape Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Figure 2-38: Secondary MDC 2, Tape Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Figure 2-39: Primary MDC 1, Tape Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
HR85-8500-A
Cray Private
1
Tiered Adaptive Storage™ Hardware Guide
Figure 2-40: Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Figure 2-41: ETHSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Figure 2-42: ETHSW1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 2-43: FCSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 2-44: FCSW1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 2-45: CIMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 2-46: Secondary MDC 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 2-47: Primary MDC 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Figure 2-48: Secondary DM 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Figure 2-49: Secondary DM 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Figure 2-50: KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Figure 2-51: Administrative Block Storage (ABS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 2-52: Data Cache Block Storage (DCBS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 2-53: Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 2-54: ETHSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 2-55: ETHSW1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 2-56: FCSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 2-57: FCSW1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 2-58: IBSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Figure 2-59: IBSW2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 2-60: CIMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 2-61: Secondary MDC 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 2-62: Primary MDC 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 2-63: Secondary DM 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 2-64: Primary DM 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 2-65: KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 2-66: Administrative Block Storage (ABS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Figure 2-67: Data Cache Block Storage (DCBS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 2-68: Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 2-69: TAS Architecture - CIMS to Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 2-70: TAS Architecture - Ethernet to MDC and DM Nodes . . . . . . . . . . . . . . . . . . . . 29 Figure 2-71: TAS Architecture - MDC/DM Node Connections . . . . . . . . . . . . . . . . . . . . . . . 29 Figure 2-72: TAS Architecture - MDC/DM Nodes to Switches, DCBS, and DAST . . . . . . . 30 Figure 3-1: TAS System Cabinet Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 3-2: TAS Shipping Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 3-3: TAS System Floor Cutout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Figure 4-1: System Board Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Figure 4-2: CIMS Front Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Figure 4-3: CIMS Rear Panel Features and Indicators, . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Figure 4-4: CIMS Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Figure 4-5: MDC and DM Node System Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Figure 4-6: MDC and DM Front Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . 46 Figure 4-7: MDC and DM Rear Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . 47
2
Cray Private
HR85-8500-A
Contents
Figure 4-8: MDC and DM Inside the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 Figure 4-9: CIMS, MDC, and DM LCD Panel Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 Figure 4-10: CIMS, MDC, and DM Hard Drive Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . .50 Figure 4-11: CIMS, MDC, and DM AC Power Supply Status Indicators . . . . . . . . . . . . . . . .50 Figure 4-12: CIMS, MDC, and DM DC Power Supply Status Indicators . . . . . . . . . . . . . . . .51 Figure 4-13: Rear View of an 8-port Rack Mount LCD KVM Assembly . . . . . . . . . . . . . . . .52 Figure 4-14: ABS Controller-Drive Tray - Front View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 Figure 4-15: ABS Controller-Drive Tray - Rear View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55 Figure 4-16: ABS Right-Rear Subplate with No Host Interface Card . . . . . . . . . . . . . . . . . . .55 Figure 4-17: ABS Right-Rear Subplate with a Two-Port 12-Gb/s SAS Host Interface Card .56 Figure 4-18: ABS Host Channels on the Controller with a Two-Port SAS HIC - Rear View .56 Figure 4-19: ABS LEDs on the Left End Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Figure 4-20: ABS LEDs on the Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 Figure 4-21: ABS LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . . . . . . . .59 Figure 4-22: ABS LEDs on the Controller Canister Two-Port SAS Host Interface Card Subplates* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 Figure 4-23: ABS LEDs on the Power-Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62 Figure 4-24: DCBS (FC) Controller-Drive Tray - Front View . . . . . . . . . . . . . . . . . . . . . . . .63 Figure 4-25: DCBS (FC) Controller-Drive Tray - Rear View . . . . . . . . . . . . . . . . . . . . . . . . .63 Figure 4-26: DCBS (FC) Host Channels on the Controller-Drive Tray - Rear View . . . . . . .64 Figure 4-27: DCBS (FC) LEDs on the Left End Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 Figure 4-28: DCBS (FC) LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . .66 Figure 4-29: DCBS (FC) Right-Rear Subplate with an FC Host Interface Card . . . . . . . . . . .68 Figure 4-30: DCBS (FC) LEDs on the Power Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 Figure 4-31: DCBS (IB) Host Channels on the Controllers with InfiniBand HIC - Rear View . 71 Figure 4-32: DCBS (IB) LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . .72 Figure 4-33: DCBS (IB) LEDs on the Controller Canister InfiniBand Host Interface Card Subplates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 Figure 4-34: DCBS (IB) LEDs on the Left Bezel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75 Figure 4-35: DCBS (IB) LEDs on the Power Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 Figure 4-36: DCBS (IB) LEDs on the Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77 Figure 4-37: ABS Drive Channel Ports on the Controller-Drive Tray - Rear View . . . . . . . .81 Figure 4-38: ABS Controller with Base Host Channel Ports (CH1 and CH2) . . . . . . . . . . . . .81 Figure 4-39: Drive Channel Ports on the Controller-Drive Tray . . . . . . . . . . . . . . . . . . . . . . .81 Figure 4-40: DCBS (IB) Drive Channel Ports on the Controller-Drive Tray . . . . . . . . . . . . .82 Figure 4-41: DCBS (IB) Controller with Labeled Drive Channel Ports (CH1 and CH2) . . . .82 Figure 4-42: ABS Drive Tray - Front View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83 Figure 4-43: ABS Drive Tray - Rear View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83 Figure 4-44: ABS Drive Tray with AC Power Option - Rear View . . . . . . . . . . . . . . . . . . . .83 Figure 4-45: ABS LEDs on the ESM Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84 Figure 4-46: ABS LEDs on the AC Power-Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 Figure 4-47: DCBS DE6600 Drive Tray - Rear View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 Figure 4-48: DCBS Maximum Capacity E-Series Array Configuration using DE6600 Shelves
HR85-8500-A
Cray Private
3
Tiered Adaptive Storage™ Hardware Guide
86 Figure 4-49: DCBS LEDs on the Left Bezel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Figure 4-50: DCBS LEDs on the ESM Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Figure 4-51: DCBS LEDs on the Power Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Figure 4-52: DCBS LEDs on the Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Figure 4-53: DCBS LEDs on the Drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Figure 4-54: DCBS LEDs on the Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Figure 4-55: Mellanox SX6036 Switch System Power Side Panel . . . . . . . . . . . . . . . . . . . . . 96 Figure 4-56: Mellanox SX6036 Switch System Connector Side Panel (Internally Managed) 96 Figure 4-57: Mellanox Switch Status LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Figure 4-58: Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Figure 4-59: Power Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Figure 4-60: Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Figure 4-61: PS Unit Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Figure 4-62: Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Figure 4-63: Identifier LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Figure 4-64: Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Figure 4-65: QLogic 5802V Chassis Controls and LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Figure 4-66: QLogic 5802V Fibre Channel Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Figure 4-67: QLogic 5802V Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Figure 4-68: QLogic 5802V Ethernet Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Figure 4-69: QLogic 5802V Serial Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Figure 4-70: Model 5802V Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Figure 4-71: PowerConnect 5548 Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Figure 4-72: Dell PowerConnect 5548 Front LED Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Figure 4-73: Giga Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4
Cray Private
HR85-8500-A
Contents
List of Tables Table 1-1: TAS Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Table 3-2: TAS System Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Table 3-3: Access Route Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Table 3-4: TAS Air Quality, Temperature, and Relative Humidity Levels . . . . . . . . . . . . . . .36 Table 3-5: Electrical Service Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 Table 4-1: CIMS Front Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 Table 4-2: CIMS Rear Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 Table 4-3: MDC and DM Front Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . .46 Table 4-4: MDC and DM Rear Panel Features and Indicators . . . . . . . . . . . . . . . . . . . . . . . . .47 Table 4-5: CMS, MDC, and DM Drive-Status Indicator Patterns . . . . . . . . . . . . . . . . . . . . . .50 Table 4-6: CIMS, MDC, and DM Power Indicator Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . .51 Table 4-7: KVM Monitor Push-button Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Table 4-8: KVM Hotkey Commands and Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Table 4-9: ABS LEDs on the Left End Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Table 4-10: ABS LEDs on the Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 Table 4-11: ABS Drive State LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 Table 4-12: ABS LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . . . . . . . .59 Table 4-13: ABS LEDs on the Controller Canister Two-Port SAS Host Interface Card Subplates* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 Table 4-14: ABS LEDs on the Power-Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62 Table 4-15: DCBS (FC) LEDs on the Left End Cap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 Table 4-16: DCBS (FC) LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . . .66 Table 4-17: DCBS (FC) LEDs on the Controller Canister Fibre Channel Host Interface Card Subplates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 Table 4-18: DCBS (FC) LEDs on the Power Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 Table 4-19: DCBS (IB) LEDs on the Controller Canister Main Faceplate . . . . . . . . . . . . . . .72 Table 4-20: DCBS (IB) LEDs on the Controller Canister InfiniBand Host Interface Card Subplates* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74 Table 4-21: DCBS (IB) LEDs on the Left Bezel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75 Table 4-22: DCBS (IB) LEDs on the Power Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 Table 4-23: DCBS (IB) LEDs on the Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77 Table 4-24: ABS and DCBS LED Symbols and General Behavior . . . . . . . . . . . . . . . . . . . . .78 Table 4-25: ABS LEDs on the ESM Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84 Table 4-26: ABS LEDs on the Power-Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 Table 4-27: DCBS LEDs on the Left Bezel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 Table 4-28: DCBS LEDs on the ESM Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88 Table 4-29: DCBS LEDs on the Power-Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89 Table 4-30: DCBS LEDs on the Fan Canister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 Table 4-31: DCBS LEDs on the Drive Tray Drawer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 Table 4-32: DCBS Drive State Represented by the LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 Table 4-33: LEDs on the Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
HR85-8500-A
Cray Private
1
Tiered Adaptive Storage™ Hardware Guide
Table 4-34: DCBS Drive State Represented by the LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Table 4-35: DCBS LED Symbols and General Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Table 4-36: Mellanox SX6036 System Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Table 4-37: Switch Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Table 4-38: Fan Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Table 4-39: PS Unit Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Table 4-40: Bad Port LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Table 4-41: Port Connector Physical and Logical Link Assignments for Ethernet Mode . . 101 Table 4-42: Connector Physical and Logical Link Assignments for IB Mode . . . . . . . . . . . 101 Table 4-43: QLogic 5802V Chassis Controls and LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Table 4-44: QLogic 5802V Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Table 4-45: QLogic 5802V Power Supply ED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Table 4-46: QLogic 5802V Power Supply LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Table 4-47: Dell PowerConnect 5548 Front Panel LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Table 4-48: Giga Ports on non-POE Enabled Devices LEDs . . . . . . . . . . . . . . . . . . . . . . . . 110 Table 4-49: HDMI (Stacking) Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Table 4-50: SFP Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Table 5-51: Storage Controller Tray ID Failure Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Table 5-52: Controller Drive Tray Sequence Code Definitions . . . . . . . . . . . . . . . . . . . . . . 114 Table 5-53: Supported Diagnostic Lock-Down Codes on the Tray ID Indicator Display . . 115 Table 5-54: Tray ID Indicator Display Uses Cases with Repeating Sequences . . . . . . . . . . 116 Table 5-55: Supported Diagnostic Codes on the ESM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Table 5-56: Sequence Code Definitions for the 5400 Controller-Drive Trays . . . . . . . . . . . 118 Table 5-57: Supported Diagnostic Lock-Down Codes on the Tray ID Indicator Display . . 119 Table 5-58: Diagnostic Code Sequences for the E5400 Controller-Drive Trays . . . . . . . . . 120 Table 5-59: Supported Diagnostic Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
2
Cray Private
HR85-8500-A
Introduction [1] About this Document Purpose and Scope This document is intended for Cray employees who need hardware overview and configuration information for Cray TAS systems. This document provides the following information: Chapter 1 - provides an high-level introduction Chapter 2 - provides an architectural overview of the system Chapter 3 - provides site planning and system specifications Chapter 4 - describes the hardware implementation of the storage system Chapter 5 - details block storage tray ID codes for the ABS and DCBS (FC and IB)
Definition of a Cray TAS System A Cray TAS system is a configuration of server, disk, tape and network hardware that runs the Versity Storage Manager (VSM) software, a Linux version of SAM-QFS that has been developed by Versity, Inc. SAM/QFS was originally developed by LSC, Sun and now Oracle for Solaris. VSM is a port of the open source version of SAM-QFS to Linux. Cray will supply first and second level support for VSM and Versity will provide third level support. Cray will supply first, second and third level support for the rest of TAS. Cray will sell and support TAS for use on XC30, XE6, CCS and other non-Cray systems running Linux.
HR85-8500-A
•
The Cray TAS system has two major functional capabilities which are:
•
A POSIX shared file system with good support for small files
•
An archive capability that manages data between disk and/or tape tiers via a policy engine capable of managing up to 4 copies of files
Cray Private
1
Tiered Adaptive Storage™ Hardware Guide
TAS has at its core: •
A metadata management system (MMS)
•
A Gigabit Ethernet (GigE) or 10 Gigabit Ethernet (10GbE) metadata network VLAN
•
A GigE management network (which is usually a separate VLAN on the metadata network)
•
An 8Gb/s Fiber Channel(FC8) or IB QDR storage network (change to FC16 and IB FDR in late 2014)
•
An optional FC8 archive network (change to FC16 in late 2014)
•
Connectivity to a public network
•
A fast data disk cache
•
An optional capability to support one or more disk or tape archive tiers
•
A policy engine that controls movement of data between storage tiers
Cray will supply and support TAS with redundant (default) and non-redundant networks and will support connectivity to a customer's incumbent networks and storage provided they follow the TAS rule conventions documented in the configuration rules
Product Naming Conventions Product naming conventions for standard configurations are based on customer requirements. In this guide, the following terminology refers to the standard configuration. Table 1-1: TAS Naming Conventions
2
TAS Naming
TAS Abbreviation Naming
Tiered Adaptive Storage system
TAS system
Cray Integrated Management System
CIMS
Dell 620
Metadata Controller
MDC
Dell 720
Data mover node
DM
Dell 720
Data Archive Storage Tape
DAST
Tape library
Data Archive Storage Disk
DASD (dsk_archive0 -dsk_archiveX)
Any storage sold (Cray or customer owned)
Data Cache Block Storage
DCBS (dc0000 -- dc000x)
NetApp 5400 or NetApp 5500
Metadata Block Storage
MDBS (md0)
NetApp 2700
Cray Private
Product
HR85-8500-A
Introduction [1]
Table 1-1: TAS Naming Conventions (continued) TAS Naming
TAS Abbreviation Naming
Product
Administrative Block Storage
ABS (adm0 FS = tas_admin)
NetApp 2700
File System Storage
tasfs1
NetApp 5500
Archive storage
DAST or DASD Data Archive Storage Tape Data Archive Storage Disk
Tape library Any storage sold (Cray or customer owned)
Additional Information Sources The following TAS document are available under the Data Management Platform (DMP) section of CrayDoc: •
Installing Cray Tiered Adaptive Storage (TAS) Software, S-2539-10
•
Tiered Adaptive Storage Software Administrator’s Guide, S-2540-10
All current Dell, NetApp, SpectraLogic, and Versity documents, along with information on virtual box server, lustre HSM connector, and TAS administrative information is located on the TAS wiki.
HR85-8500-A
Cray Private
3
Tiered Adaptive Storage™ Hardware Guide
Safety Precautions and Hazards Failure to observe the warnings and cautions throughout this document could result in death or serious injury.
Warning: All procedures in this document must be performed by Cray trained service providers or authorized site personnel. Failure to observe this warning could result in death or serious injury.
Warning: Remove watches and jewelry when installing this equipment. Keep fingers and conductive tools away from high-voltage areas as well as the highcurrent areas on the system. High-voltage areas are indicated with high-voltage warning labels.
Warning: Wait for the system to power down completely before you touch any components associated with the high-voltage circuits. Failure to do so will result in death or serious injury.
Warning: Invisible laser radiation ma be emitted from the end of the unterminated fiber cable or connector. Do not view directly with optical instruments. Viewing the laser output with certain optical instruments (for example, eye loupes, manifiers, and microscopes) within a distance of 100 mm may pose an eye hazard.
Caution: Remove all tools from the system cabinets and replace all covers and panels removed from the system during servicing. Failure to do so could result in serious injury or equipment damage.
Caution: Halt the operation system before removing power from the system. Failure to do so can result customer data loss or corruption.
4
Cray Private
HR85-8500-A
Introduction [1]
Caution: Observe electrostatic discharge (ESD) precautions during the entire installation process. Required apparel includes an ESD smock and an ESD wrist strap. Failure to do so can result in damage to the equipment.
Caution: Other recommended safety apparel are gloves and certified reinforced-toe safety shoes.
Caution: Some procedures require the assistance of qualified personnel wearing proper safety apparel.
HR85-8500-A
Cray Private
5
Tiered Adaptive Storage™ Hardware Guide
6
Cray Private
HR85-8500-A
System Architecture [2] Software Architecture For more information on the software architecture, refer to the Cray Tiered Adaptive Storage (TAS) Administrator’s Guide. S-2540-10, available from CrayDoc.
Hardware Configurations TAS Gateway System The TAS gateway system is the minimum supported hardware configuration. It comprises of the Cray Integrated Management Services (CIMS) node, metadata controller (MDC) nodes, management Ethernet switch, and administrative block storage (ABS). Figure 2-1 shows the TAS Gateway minimum hardware configuration, and Figure 2-2 through Figure 2-7 show the TAS Gateway rear view and cabling.
HR85-8500-A
Cray Private
7
Tiered Adaptive Storage™ Hardware Guide
Figure 2-1: TAS Gateway Minimum Hardware Configuration Cray Integrated Maintence Sevices (CIMS) node(s) eth0
eth2
eth1
DRAC ipmi0 site-admin-net
site-admin-net
ipmi-net
esmaint-net
Bright
metadata-net VLAN 4
site-admin-net VLAN 3
ipmi-net VLAN 2
esmaint-net VLAN 1
Ethernet switch Customer network customer-net
10.141 10.148
10GbE
metadata-net site-admin-net ipmi-net esmaint-net 10.50
failover-net
eth3 eth0
DRAC
eth1 eth2 eth5 eth4 10GbE Primary MDC Node
eth3 eth0 DRAC eth1 eth2 5eth5 hte eth4 10GbE Secondary MDC Node Port B
Port A
Port A
Port B SAS
SAS Cntrl B
Cntrl A
Disks or SSDs
Administrative Storage (MABs)
Figure 2-2: TAS Gateway (Front View)
Ethernet Switch CIMS
EST
EST
MDCs EST
ABS
8
Cray Private
8
16
9
17
HR85-8500-A
System Architecture [2]
Figure 2-3: ETHSW1, TAS Gateway ETHSW1 VLAN1
VLAN2
esmaint-net 1
3
5
7
9
11
13
VLAN3
ipmi-net
15
17
19
21
23
25
27
29
VLAN4
site-admin-net metadata-net 31
33
35
37
39
41
43
45
LNK /ACT
FDX
DIAG
RPS
FAN
TEMP
PWR
47 /HDX COMBO PORTS
RESET LNK/ACT
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
LNK/ACT
LNK/ACT
LNK/ACT
48
Figure 2-4: CIMS, TAS Gateway CIMS 1 1
750 W
2 1
2
3
750 W
4
{
iDRAC
eth3 failover-net to second CIMS node eth2 to VLAN 2 - ipmi-net
To KVM
eth1 to VLAN 3 - site-admin-net iDRAC to VLAN 2 - ipmi-net
eth0 to VLAN 1 - esmaint-net
Figure 2-5: Secondary MDC 2, TAS Gateway Slot 1, port 2 to ABS Cntr B Port 2
10GbE to Customer Network
{
Slot 1, port 1 to ABS Cntr A Port 2
4
1
ST
Slot 2, ports 1 and 2 to customer FC network Slot 3, ports 1 and 2 to customer IB network
Slot 7, ports 1 and 2 to customer FC network
Secondary MDC 2
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC1
iDRAC to VLAN 2 - ipmi-net
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-6: Primary MDC 1, TAS Gateway Slot 1, port 2 to ABS Cntr B Port 1
10GbE to Customer Network
{
Slot 1, port 1 to ABS Cntr A Port 1
4
1
ST
Slot 2, ports 1 and 2 to customer FC network Slot 3, ports 1 and 2 to customer IB network
Slot 7, ports 1 and 2 to customer FC network
Primary MDC 1
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC2
iDRAC to VLAN 2 - ipmi-net
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-7: Administrative Block Storage (ABS), TAS Gateway Administrative Block Storage Ch 1 Lnk
Ch 2
Lnk
Lnk
Port 1
Port 2
Lnk Lnk Port 1
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Lnk
Lnk Lnk
Port 1
Lnk
Port 2
Lnk
Lnk
Lnk
AC
DC
Cntrl A Ethernet Port 1 to esmaint-net
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Port 1
Port 2
Lnk Lnk
Lnk
Drive Expansion
ID/ Diag Lnk
SAS Cntrl A Ch2 to MDC2 SAS Cntrl A Ch1 to MDC1
HR85-8500-A
Ch 2
Lnk Lnk
Drive Expansion Lnk
DC
Ch 1
Lnk Lnk
Lnk
ID/ Diag
Lnk Lnk
Lnk
AC
SAS Cntrl B Ch2 to MDC2 SAS Cntrl B Ch1 to MDC1 Cntrl B Ethernet Port 1 to esmaint-net
Cray Private
9
Tiered Adaptive Storage™ Hardware Guide
TAS Connector The TAS Connector contains three software services, which are required for proper operation of the system. The three services are RobinHood, Migration Manager , and Migration Agent. RobinHood and Migration Manager are co-located on a single OS node called the Manager Nodes. There are two manager nodes in the HA failover architecture; the manager nodes run on two Dell 720’s. The two manager nodes connect to the NetApp E2700, otherwise known as the administrative block storage (ABS). The NetApp E2700 has 4 600GB disk drives, RAID 10, owned by controller B. RobinHood and Migration Manager nodes have no FC capability. The nodes can only connect to the TAS HSM by InfiniBand or SAS. Refer to Figure 2-8 through Figure 2-12 for the TAS Connector Manager (TCM) Lustre PCIe slots and administrative block storage (ABS) connections. The final service, the Migration Agent, exists on a DM node in the TAS HSM. The Migration Agent connects the TAS HSM to the Lustre client through an InfiniBand connection. If a Migration Agent fails, the Migration Manager will restart the work on another agent. TAS Connector 1.0 HA failover is manually initiated. All services on RobinHood and the Migration Manager must be shutdown in case a user initiates a failover. If a system failure occurs, a secondary node must be brought up so data stores can be checked. Figure 2-8: TAS Connector Manager Node with IB Lustre TCM Node with IB Lustre 1
4
SAS ST
2
IB Disk Lustre
5
6 7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-9: TAS Connector Agent on a TAS DM with IB Lustre and IB TAS TAS Connector Agent on a TAS DM with IB Lustre and IB TAS 4
1
ST
2
IB Disk Lustre
5
6 7 2
1 3
IB Disk TAS HSM 750 W
1
2
3
750 W
4
iDRAC
Figure 2-10: TAS Connector Manager SAS Connections, Secondary TCM2 Port 2 to ABS Cntr B Port 2 Port 1 to ABS Cntr A Port 2
Secondary TCM2 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
10
Cray Private
HR85-8500-A
System Architecture [2]
Figure 2-11: TAS Connector Manager SAS Connections, Primary TCM 1 Port 2 to ABS Cntr B Port 1
Primary TCM1
Port 1 to ABS Cntr A Port 1
4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-12: TAS Connector Manager SAS Connections, Administrative Block Storage Administrative Block Storage Ch 1 Lnk
Ch 2
Lnk
Lnk
Port 1
Port 2
Lnk Lnk Port 1
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Lnk
Lnk Lnk
Lnk
Lnk
Lnk
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Port 2
DC
Lnk Lnk
Lnk
Drive Expansion
ID/ Diag Lnk
AC
Cntrl A Ethernet Port 1 to esmaint-net
Port 1
Lnk Lnk Port 1
SAS Cntrl A Ch2 to TCM2 SAS Cntrl A Ch1 to TCM1
HR85-8500-A
Ch 2
Lnk
Drive Expansion Lnk
DC
Ch 1
Lnk
Lnk
ID/ Diag
Lnk Lnk
Lnk
AC
SAS Cntrl B Ch2 to TCM2 SAS Cntrl B Ch1 to TCM1 Cntrl B Ethernet Port 1 to esmaint-net
Cray Private
11
Tiered Adaptive Storage™ Hardware Guide
Full TAS Configuration Cray Tiered Adaptive Storage (TAS) systems are flexible in hardware configurations and architecture. A full TAS configuration includes all disks, MDC’s, and DM’s. A full TAS system could include a CIMS, two metadata controllers (MDC’s) and data mover nodes (DM’s), management switches (Ethernet, FC, and IB), administrative block storage (ABS), and data cache block storage (DCBS). A tape library is optional because the customer can provide their own. Figure 2-13 and Figure 2-14 show example front and review views of a full TAS system. These examples include a tape library. The tape library is in the TAS cabinet for this example, but the tape library can be located in another cabinet. Product naming conventions can be found in “Product Naming Conventions” on page 2.
12
Cray Private
HR85-8500-A
System Architecture [2]
Figure 2-13: Full TAS System (Front View) 42
42
41
LOCATOR
DC IN
RPS 12V,15A
LOCATOR
DC IN
RPS 12V,15A
41
42
Zoned Ethernet switches
42 100V-240VAC, 1.6A
40
40
41
41 100V-240VAC, 1.6A
39
39
38
38
40
40
FC switches 39
39
12C !
37
IB switches
! !
37 38
12C !
36
37
CIMS
!
Mellanox
38
!
Mellanox
! !
36
35
35
36
EST
34
34
33
33
32
32
31
31
35
37
36
35
34
34
MDCs 33
32
0
30
1
2
3
4
5
6
7
8
9
10
11
12
13
14
30
29
29
30
DMs
0
28
1
2
3
4
5
6
7
8
9
10
11
12
13
14
28
27
27
26
26
25
25
24
24
28
27
26
KVM
25
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
MABS
22
23
23
22
22
23
21
6
9 12
2
5
8 11
1
2
3
4
5
6
1
3
4
7
6
9 12
2
5
8 11
1
2
3
4
5
6
1
4
7
22
10
21 7
8
9
10 11 12
7
8
9
10 11 12
7
8
9
10 11 12
30
29
28
27
26
25
24
23
24
31
15
EST
29
32
15
EST
31
33
24
23
22
Front
3
20
21
DCBS
10
20
Front
3
1
2
3
4
5
6
6
2
5
1
4
3
1
2
3
4
5
6
21
9 12
8 11
7
10
Front
19
19
20
20
9 12
2
5
6
1
4
6
9 12
2
5
8 11
6
1
4
8 11
7
7
10
7
8
9
10 11 12
Front
18
3
19
1
2
3
4
5
10
18 7
8
9
19
10 11 12
Front
17
17
16
16
15
15
1514
14
13
13
12
12
1211
11
10
10
9
9
98
8
9
87
7
8
6
6
65
5
6
54
4
5
3
3
4
32
2
3
21
1
2
18
17
16
14
13
Tape
11
10
7
4
1
18
17
16
15
14
13
12
11
10
7
1
220VAC PDU North America: NEMA L6-30R International: IEC 309 32A 2P+E
HR85-8500-A
Cray Private
13
Tiered Adaptive Storage™ Hardware Guide
Figure 2-14: Full TAS System (Rear View) 42
42
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
41
43
45
tus Sta R PW
1
LNK 47 ACT
41
Reset
42
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
48
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
41
43
45
LNK 47 ACT
ACT
1
LNK
ACT
Stack No. LNK
2
Stacking HDMI
1
40
Reset
41
LNK
2
4
6
8
10
12
14
16
18
20
22
24
26
2-3>
0-1>
28
30
32
34
4-5>
36
38
40
42
44
46
48
8-9>
6-7>
ACT
1
LNK
Stacking HDMI
ACT
Stack No. LNK
2
20
0
41
M RPS Fan
42
ACT
SFP+
2
tus Sta R PW
Zoned Ethernet switches
LNK
1
40
M RPS Fan
41
ACT
SFP+
2
22
39
39
40
FC switches
SANbox 5800 10-11>
12-13>
14-15>
16-17>
18-19>
0-1>
2-3>
4-5>
6-7>
8-9>
0
21
23
20
22
40
38
38
39
39 SANbox 5800
21 10-11>
12-13>
23
18-19>
16-17>
14-15>
SX6036
MGT 1
37 38
3
2
4
5
7
6
8
9
10
11
13
12
15
14
17
16
19
18
21
20
23
22
25
24
26
27
29
28
31
30
33
32
34
35
CONSOLE
36
37
PS1
38
PS2
IB switches
UID RST
SX6036
MGT 1
36
37
3
2
4
5
7
6
8
9
10
11
13
12
15
14
17
16
19
18
21
20
23
22
25
24
26
27
29
28
31
30
33
32
34
35
CONSOLE
36
36
PS1 PS2
37
UID RST
CIMS
35
35
36
Gb 1
Gb 2
Gb 3
36
Gb 4
34
34
33
33
32
32
31
31
35
35
34
34
MDCs 33
32
0
30
1
2
3
4
5
6
7
8
9
10
11
12
13
14
30
29
29
30
DMs
0
28
1
2
3
4
5
6
7
8
9
10
11
12
13
14
28
27
27
26
26
25
25
28
27
26
100-240V~, 0.5A, 50/60 Hz
KVM
24
25
1
3
5
7
2
4
6
8
24
LAN
23
Lnk
24
Ch 1
Ch 2
Lnk
Lnk
Port 1
Port 2
Lnk
MABS
Lnk
22
Port 1
Lnk
Port 2
Lnk
Ch 3
Ch 4
Lnk
23
29
28
27
26
25
24
Lnk
Drive Expansion Lnk
30
Lnk
ID/ Diag
23
31
15
EST
29
32
15
EST
31
33
22
Lnk
23
2
21
21
1
DC
AC
PSU A
2
3
4
PSU B
1
5
2
1
6
6
9 12
5
8
11
4
7
10
22
PSU A
7
8
9
PSU B
3
22
10 11 12
Front
Lnk
Lnk
Lnk
Lnk
3
2
ESM B
2
3
4
5
Ch 1
6
Ch 2
1
2
3
Port 2
4
Lnk
5
Lnk
Lnk
Lnk
2
1
6 Ch 1
9 12
8
11
7
10 IB Host Card
Front
3
Port 1
1
6
5
4
Act
Lnk 7
Act
Lnk
ESM A
8 Act9
8
7
10
7 Lnk
IB Host Card
8
9
19 ESM B
ESM A
3
4
5
3
6
9 12
2
5
8
11
7
10
1
6
4
2
7
8
9
AC
PSU B
18
PSU A
PSU A 3
2
2
20
10 11 12
Front
1
DC
PSU B
19
1
21
10 11 12
Act
ESM A
2
20
11
CH 2
1
18
10 11 12
9 12
Front
Ch 2
ID/ Diag
ESM B
Lnk
CH 1
19
20
6
5
4
CH 2
ID/ Diag
CH 1
DCBS
Port 2
ESM A
1
ESM B
Port 1
20
21
3
4
5
6
1
6
9 12
5
8
11
4
7
10
7
8
9
19
10 11 12
Front
17
17
16
16
15
15
18
17
16
Fibre B
14
Drive Pwr
Fibre A
Status
Drive Pwr
Status
Fibre A
13
Fibre B
11
Drive Pwr
Fibre A
Status
Drive Pwr
Status
Fibre A
10
11
Fibre B
Drive Pwr
Fibre A
Status
Drive Pwr
Status
Fibre A
14
13
12
11
10
8
9
7
8
6
6
Fibre B
7
Fibre B
Drive Pwr
Fibre A
Status
Status
Fibre A
Drive Pwr
65
54
15
Fibre B
98
7
10
9
9
87
16
Fibre B
1211
10
13
12
12
13
Tape
17
Fibre B
14
15
14
18
5
6
4
5
3
3
32
2
3
21
1
2
4
1
4
1
220VAC PDU North America: NEMA L6-30R International: IEC 309 32A 2P+E
14
Cray Private
HR85-8500-A
System Architecture [2]
Full TAS Basic Configuration Cabling The following figures show the full TAS configuration cabling based on the full TAS configuration examples. The cabling will vary with every TAS system due to varying configurations. For any TAS configuration, if there are more than 12 tape drives, add FCSW3 and FCSW4. If there are more than 2 data mover (DM) nodes, add FCSW3 and FCSW4.
MDC PCIe Slot Layout Figure 2-15: MDC with FC Data Cache MDC with FC Data Cache 1
SAS MDT
1
FC Tape
1
3
FC Disk
1
ST
2
4 2 2
0
10GigE - Customer Net
1
2
FC Disk
1
2
FC Tape
1
5
6 7 2
1 2 750 W
1
2
3
750 W
4
iDRAC
Figure 2-16: MDC with IB Data Cache MDC with IB Data Cache 1
3
1
SAS MDT
1
FC Tape
1
IB Disk
4
6
2
ST
2
0
2
5
1
10GigE - Customer Net
2
FC Tape
7 1
2
1 2 750 W
1
2
3
750 W
4
iDRAC
DM PCIe Slot Layout Figure 2-17: DM with FC Data Cache DM with FC Data Cache 4
1
2
ST
2 3
1
FC Disk
0
1
FC Disk
5
10GigE - Customer Net
6 1
7 2
1 2 750 W
1
2
3
750 W
4
iDRAC
Figure 2-18: DM with IB Data Cache DM with IB Data Cache 4
1
ST
2 3
1
IB Disk
0
6
5
1
10GigE - Customer Net
7 2
1 2 750 W
1
2
3
750 W
4
iDRAC
HR85-8500-A
Cray Private
15
Tiered Adaptive Storage™ Hardware Guide
Metadata Connections Figure 2-19: Secondary MDC 2 Port 2 to ABS Cntr B Port 2 Port 1 to ABS Cntr A Port 2
Secondary MDC 2 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-20: Primary MDC 1 Port 2 to ABS Cntr B Port 1 Port 1 to ABS Cntr A Port 1
Primary MDC 1 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-21: Administrative Block Storage Administrative Block Storage Ch 1
Ch 2
Lnk
Lnk
Lnk
Port 1
Port 2
Lnk Lnk Port 1
Lnk
Port 2
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
DC
Lnk Lnk
Lnk
Ch 2
Lnk
Lnk
Lnk
Port 1
Port 2
Lnk Lnk
Drive Expansion Lnk
Ch 1
Lnk
Lnk
ID/ Diag
Port 1
Port 2
Lnk
Lnk
Lnk
Ch 3
Ch 4
DC
SAS Cntrl A Ch2 to MDC2 SAS Cntrl A Ch1 to MDC1
Ch 5
Ch 6
Lnk Lnk
Lnk
Drive Expansion Lnk
AC
Lnk Lnk
ID/ Diag Lnk Lnk
Lnk
AC
SAS Cntrl B Ch2 to MDC2 SAS Cntrl B Ch1 to MDC1 Cntrl B Ethernet Port 1 to esmaint-net
Cntrl A Ethernet Port 1 to esmaint-net
FC Data Cache Figure 2-22: FCSW2, Data Cache To Primary and Secondary MDC, Slot 3, Port 2 and Slot 6, Port 1
FCSW2 MDC/DM to FCSW - Data Cache
To Primary and Secondary DM Slot 3, Port 2 and Slot 6, Port 1 0-1> > 0
SANbox 5800 10-11>
12-13>
14-15>
16-17>
To DCBS Cntr A, Port 2
Data Cache to FCSW MDC/DM
To DCBS Cntr B, Port 2 20
22
21
23
18-19>
Figure 2-23: FCSW1, Data Cache To Primary and Secondary MDC, Slot 3, Port 1 and Slot 6, Port 2
FCSW1 MDC/DM to FCSW - Data Cache
To Primary and Secondary DM Slot 3, Port 1 and Slot 6, Port 2 0-1> > 0
SANbox 5800 10-11>
16
12-13>
Cray Private
14-15>
16-17>
To DCBS Cntr A, Port 1
Data Cache to FCSW MDC/DM
To DCBS Cntr B, Port 1 20
22
21
23
18-19>
HR85-8500-A
System Architecture [2]
Figure 2-24: Secondary MDC 2, FC Data Cache Port 2 - MDC/DM to FCSW1 - Data Cache Port 1 - MDC/DM to FCSW2 - Data Cache
Secondary MDC 2 4
1
ST
2
Port 2 - MDC/DM to FCSW2 - Data Cache Port 1 - MDC/DM to FCSW1 - Data Cache
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-25: Primary MDC 1, FC Data Cache Port 2 - MDC/DM to FCSW1 - Data Cache Port 1 - MDC/DM to FCSW2 - Data Cache
Primary MDC 1 4
1
ST
Port 2 - MDC/DM to FCSW2 - Data Cache Port 1 - MDC/DM to FCSW1 - Data Cache
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-26: Secondary DM 2, FC Data Cache Port 2 - MDC/DM to FCSW1 - Data Cache Port 1 - MDC/DM to FCSW2 - Data Cache
Secondary DM 2 4
1
ST
Port 2 - MDC/DM to FCSW2 - Data Cache Port 1 - MDC/DM to FCSW1 - Data Cache
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-27: Primary DM 1, FC Data Cache Port 2 - MDC/DM to FCSW1 - Data Cache Port 1 - MDC/DM to FCSW2 - Data Cache
Primary DM 1 4
1
ST
Port 2 - MDC/DM to FCSW2 - Data Cache Port 1 - MDC/DM to FCSW1 - Data Cache
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-28: DCBS, FC Data Cache Cntrl A, Port 2 - Data Cache to FCSW2 - MDC/DM Cntrl A, Port 1 - Data Cache to FCSW1 - MDC/DM
DCBS (FC) 2 1
DC
AC
PSU A
PSU A PSU B
PSU B
Port 1
Port 2
2
4
2
4
8
2
8
4
2
8
4 Lnk
8
Lnk
ESM A
ESM B
Ch 2 FCHost Ch 3
Ch 1
Drive Expansion
Ch 4
ESM B
ESM A
ID/Diag
Port 2
2
4
2
8
4 8
2
4
Ch 2 FCHost Ch 3
Ch 1
ESM B
2
8
4 8
Ch 4
ID/Diag
Lnk
Lnk
Drive Expansion ESM B ESM A
Port 1
ESM A 2 1
DC
AC
PSU B PSU A
PSU B PSU A
Cntrl B, Port 2 - Data Cache to FCSW2 - MDC/DM Cntrl B, Port 1 - Data Cache to FCSW1 - MDC/DM
HR85-8500-A
Cray Private
17
Tiered Adaptive Storage™ Hardware Guide
IB Data Cache Figure 2-29: IBSW2 To Primary DM 1, Slot 3, Port 2
To Secondary MDC 2, Slot 3, Port 2
To DCBS Cntr A, Port 1
IBSW2 SX6036
MGT 1
3
2
5
4
0-1>
0 7
6
9
8
2-3>
11
10
12
13
4-5>
15
14
17
16
19
18
21
20
6-7>
23
22
8-9>
25
24
27
26
20
29
28
30
31
33
32
34
22 35
CONSOLE
36
PS1
To Primary MDC 1, Slot 3, Port 2 To Secondary DM 2, Slot 3, Port 2 PS2
UID
RST
SANbox 5800
21 12-13>
10-11>
23
18-19>
16-17>
14-15>
To DCBS Cntrl B, Port 1
Figure 2-30: IBSW2 To Primary DM 1, Slot 3, Port 1
To Secondary MDC 2, Slot 3, Port 1
To DCBS Cntr A, Port 2
IBSW1 SX6036
MGT 1
2
3
4
5
6
0-1>
0 7
8
9
10
2-3>
11
12
13
15
14
4-5>
17
16
19
18
20
6-7>
21
22
23
24
25
8-9> 26
27
28
29
20 30
31
32
33
34
22 35
CONSOLE
36
PS1
To Primary MDC 1, Slot 3, Port 1 To Secondary DM 2, Slot 3, Port 1 PS2
UID
RST
SANbox 5800
21 12-13>
10-11>
16-17>
14-15>
23
18-19>
To DCBS Cntrl B, Port 2
Figure 2-31: Secondary MDC 2, IB Data Cache Secondary MDC 2 4
1
ST
2
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-32: Primary MDC 1, IB Data Cache Primary MDC 1 4
1
ST
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-33: Secondary DM 2, IB Data Cache Secondary DM 2 4
1
ST
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-34: Primary DM 1, IB Data Cache Primary DM 1 4
1
ST
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
18
Cray Private
HR85-8500-A
System Architecture [2]
Figure 2-35: Primary DM 1, IB Data Cache Cntr A, Port 1 to IBSW2 Cntr A, Port 2 to IBSW1
DCBS (IB) 2 1
DC
AC
PSU A
PSU A PSU B
PSU B
Lnk
Port 1
Port 2
Lnk
Lnk
ESM A
Lnk
Ch 1
Lnk
Ch 2
Lnk
ESM B
ID/ Diag
Ch 1
Lnk
IB Host Card
Act
Lnk
Act
Act
Lnk
Act
ESM A ESM B
Lnk
CH 2
Port 2
CH 1
Port 1
Lnk
Ch 2
Lnk
CH 2
IB Host Card
CH 1
ID/ Diag
ESM B
ESM B ESM A
ESM A 2 1
DC
PSU B
AC
PSU B PSU A
PSU A
Cntr B, Port 2 to IBSW1 Cntr B, Port 1 to IBSW2
Basic Tape Configuration The tape library cabling is determined by the site. Half the cables must go to FCSW1 and half must go to FCSW2. Figure 2-40 shows a possible tape library cabling configuration. Figure 2-36: FCSW2, Tape Configuration FCSW2 2-3>
0-1>
4-5>
8-9>
6-7>
0
SANbox 5800
20
22
21
23
10-11> >
MDC to FCSW - Tape
To Primary and Secondary MDC, Slot 2, Port 2 and Slot 7, Port 2
Tape Library to FCSW MDC/DM
Tape Library, Fibre B Port
Figure 2-37: FCSW1, Tape Configuration FCSW1 2-3>
0-1>
8-9>
6-7>
4-5>
0
SANbox 5800
20
22
21
23
10-11> >
To Primary and Secondary MDC, Slot 2, Port 1 and Slot 7, Port 1
MDC to FCSW - Tape
Tape Library to FCSW MDC/DM
Tape Library, Fibre A Port
Figure 2-38: Secondary MDC 2, Tape Configuration Port 2 - MDC to FCSW2 - Tape Port 1 - MDC to FCSW1 - Tape
Secondary MDC 2 4
1
ST
Port 2 - MDC to FCSW2 - Tape Port 1 - MDC to FCSW1 - Tape
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
Figure 2-39: Primary MDC 1, Tape Configuration Port 2 - MDC to FCSW2 - Tape Port 1 - MDC to FCSW1 - Tape
Primary MDC 1 4
1
ST
Port 2 - MDC to FCSW2 - Tape Port 1 - MDC to FCSW1 - Tape
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
HR85-8500-A
Cray Private
19
Tiered Adaptive Storage™ Hardware Guide
Figure 2-40: Tape Library Tape Library
Status
Fibre A Status
Fibre B
Fibre B
Fibre A
Fibre A Status
Status
Fibre A port to FCSW2
Status
Fibre A
Fibre A port to FCSW1
Drive Pwr
Fibre B
Fibre A port to FCSW1
Drive Pwr
Fibre B
Status
Fibre A port to FCSW2
Fibre A port to FCSW1
Drive Pwr
Fibre A
Drive Pwr
Fibre B
Fibre A Status
Fibre B
Drive Pwr
Fibre A port to FCSW2
Fibre A port to FCSW1
Drive Pwr
Fibre A
Drive Pwr
Fibre B
Fibre A Status
Fibre B
Drive Pwr
Fibre A port to FCSW2
FC Full TAS Configuration There are two types of full TAS configurations. The Fibre Channel (FC) full TAS configuration consists of two management Ethernet switches and two FC switches. The data cache block storage (DCBS) is a NetApp 5400. Figure 2-41 through Figure 2-53 show the FC full TAS configuration components and cabling. Figure 2-41 shows a second Ethernet switch; however, a second Ethernet switch is only used if the system uses more than 2 DMs.
The tape library cabling is determined by the site. Half the cables must go to FCSW1 and half must go to FCSW2. Figure 2-53 shows a possible tape library cabling configuration. Figure 2-41: ETHSW2 ETHSW2 VLAN1
VLAN2
esmaint-net 1
3
5
7
9
11
13
15
VLAN3
ipmi-net 17
19
21
23
25
27
29
VLAN4
site-admin-net metadata-net 31
33
35
37
39
41
43
45
LNK /ACT
FDX
DIAG
RPS
FAN
TEMP
PWR
47 /HDX COMBO PORTS
RESET LNK/ACT
2
20
4
6
8
10
12
14
16
18
20
22
24
Cray Private
26
28
30
32
34
36
38
40
42
44
46
LNK/ACT
LNK/ACT
LNK/ACT
48
HR85-8500-A
System Architecture [2]
Figure 2-42: ETHSW1 ETHSW1 VLAN1
VLAN2
esmaint-net 1
3
5
7
9
11
13
VLAN3
ipmi-net
15
17
19
21
23
25
27
29
VLAN4
site-admin-net metadata-net 31
33
35
37
39
41
43
45
LNK /ACT
FDX
DIAG
RPS
FAN
TEMP
PWR
47 /HDX COMBO PORTS
RESET LNK/ACT
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
LNK/ACT
LNK/ACT
LNK/ACT
48
Figure 2-43: FCSW2 FCSW2
To Primary and Secondary MDC, Slot 3, Port 2 and Slot 6, Port 2 To Primary and Secondary DM Slot 3, Port 2 and Slot 6, Port 2
To DCBS Cntr A, Port 2
Data Cache to FCSW MDC/DM
MDC/DM to FCSW - Data Cache
To DCBS Cntr B, Port 2
0-1> > 0
SANbox 5800
20
22
21
23
10-11> >
MDC to FCSW - Tape
To Primary and Secondary MDC, Slot 2, Port 2 and Slot 7, Port 2
Tape Library to FCSW MDC/DM
Tape Library, Fibre B Port
Figure 2-44: FCSW1 FCSW1
To Primary and Secondary MDC, Slot 3, Port 1 and Slot 4, Port 1 To Primary and Secondary DM Slot 3, Port 1 and Slot 4, Port 1
To DCBS Cntr A, Port 1
Data Cache to FCSW MDC/DM
MDC/DM to FCSW - Data Cache 0-1> > 0
SANbox 5800
To DCBS Cntr B, Port 1 20
22
21
23
10-11> >
MDC to FCSW - Tape
To Primary and Secondary MDC, Slot 2, Port 1 and Slot 5, Port 1
Tape Library to FCSW MDC/DM
Tape Library, Fibre A Port
Figure 2-45: CIMS CIMS 1 1
750 W
2 1
2
3
750 W
4
{
iDRAC
eth3 failover-net to second CIMS node eth2 to VLAN 2 - ipmi-net
To KVM
eth1 to VLAN 3 - site-admin-net iDRAC to VLAN 2 - ipmi-net
eth0 to VLAN 1 - esmaint-net
Figure 2-46: Secondary MDC 2 Slot 1, port 2 to ABS Cntr B Port 2 Slot 1, port 1 to ABS Cntr A Port 2
{
Slot 2, Port 1 MDC to FCSW1 - Tape
Secondary MDC 2 4
1
ST
Slot 2, Port 2 MDC to FCSW2 - Tape
Slot 6, Port 2 - MDC/DM to FCSW1 - Data Cache Slot 6, Port 1 - MDC/DM to FCSW2 - Data Cache
10GbE to Customer Network
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
iDRAC to VLAN 2 - ipmi-net
Slot 3, Port 1 - Slot 3, Port 2 MDC/DM to MDC/DM to FCSW2 - Data Cache FCSW1 - Data Cache
HR85-8500-A
eth3 failover-net to MDC0 eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Slot 7, Port 1 - Slot 7, Port 2 MDC to MDC to FCSW2 - Tape FCSW1 - Tape
Cray Private
21
Tiered Adaptive Storage™ Hardware Guide
Figure 2-47: Primary MDC 1 Slot 1, port 2 to ABS Cntr B Port 1 Slot 1, port 1 to ABS Cntr A Port 1
{
Slot 2, Port 2 MDC to FCSW2 - Tape
Slot 6, Port 2 - MDC/DM to FCSW1 - Data Cache Slot 6, Port 1 - MDC/DM to FCSW2 - Data Cache
10GbE to Customer Network
Primary MDC 1 4
1
ST
2
Slot 2, Port 1 MDC to FCSW1 - Tape
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC0
iDRAC to VLAN 2 - ipmi-net
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net
Slot 3, Port 1 - Slot 3, Port 2 MDC/DM to MDC/DM to FCSW2 - Data Cache FCSW1 - Data Cache
eth0 to VLAN 1 - esmaint-net
Slot 7, Port 1 - Slot 7, Port 2 MDC to MDC to FCSW2 - Tape FCSW1 - Tape
Figure 2-48: Secondary DM 2 Slot 3, Port 2 - MDC/DM to FCSW2 - Data Cache
Slot 6, Port 2 - MDC/DM to FCSW1 - Data Cache Slot 6, Port 1 - MDC/DM to FCSW2 - Data Cache
10GbE to Customer Network
{
Slot 3, Port 1 - MDC/DM to FCSW1 - Data Cache
Secondary DM 2 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC0 iDRAC to VLAN 2 - ipmi-net
Port 1
Port 2
Lnk
Lnk
Lnk
Ch 1
eth2 to VLAN 4 - metadata-net Lnk
Ch 2
Lnk
Act
Lnk
Act
IB Host Card
CH 2
ID/ Diag
CH 1
eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-49: Secondary DM 1 Slot 3, Port 2 - MDC/DM to FCSW2 - Data Cache
Slot 6, Port 2 - MDC/DM to FCSW1 - Data Cache Slot 6, Port 1 - MDC/DM to FCSW2 - Data Cache
10GbE to Customer Network
{
Slot 3, Port 1 - MDC/DM to FCSW1 - Data Cache
Primary DM 1 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC0 iDRAC to VLAN 2 - ipmi-net
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-50: KVM KVM 100-240V~, 0.5A, 50/60 Hz
1
3
5
7
2
4
6
8
LAN
Video esmaint-net or site-admin-net
22
Cray Private
Keyboard and Mouse
Maintenance dongle to connect KVM to another server
HR85-8500-A
System Architecture [2]
Figure 2-51: Administrative Block Storage (ABS) Administrative Block Storage Ch 1 Lnk
Ch 2
Port 1
Lnk
Lnk
Port 2
Lnk Lnk Port 1
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Lnk Lnk
Ch 2
Lnk
Lnk
Lnk
Lnk
Port 1
Port 2
Lnk Lnk
Drive Expansion Lnk
DC
Ch 1
Lnk
Lnk
ID/ Diag
Port 1
Lnk
Port 2
Lnk
Ch 3
Ch 4
Ch 5
Ch 6
AC
Lnk Lnk
Lnk
Drive Expansion Lnk
DC
SAS Cntrl A Ch2 to MDC2 SAS Cntrl A Ch1 to MDC1
Lnk Lnk
ID/ Diag
Lnk
Lnk Lnk
Lnk
AC
SAS Cntrl B Ch2 to MDC2 SAS Cntrl B Ch1 to MDC1 Cntrl B Ethernet Port 1 to esmaint-net
Cntrl A Ethernet Port 1 to esmaint-net
Figure 2-52: Data Cache Block Storage (DCBS) Cntrl A, Port 2 - Data Cache to FCSW2 - MDC/DM Cntrl A, Port 1 - Data Cache to FCSW1 - MDC/DM
DCBS (FC) 2 1
DC
AC
PSU A
PSU A PSU B
PSU B
Port 1
Port 2
2
4
2
4
8
2
8
4
2
8
4 Lnk
8
Lnk
ESM A
ESM B
Ch 2 FCHost Ch 3
Ch 1
Drive Expansion
Ch 4
ESM B
ESM A
ID/Diag
Port 1
Port 2
2
4
2
8
4 8
2
4 8
Ch 2 FCHost Ch 3
Ch 1
ESM B
2
4 Lnk
8
Lnk
Drive Expansion
Ch 4
ESM B
ID/Diag
ESM A
ESM A 2 1
DC
PSU B
AC
PSU B PSU A
PSU A
Cntrl B, Port 2 - Data Cache to FCSW2 - MDC/DM Cntrl B, Port 1 - Data Cache to FCSW1 - MDC/DM
Figure 2-53: Tape Library
HR85-8500-A
Status
Fibre A
Fibre B
Fibre B
Fibre A
Fibre A
Cray Private
Drive Pwr
Status
Fibre A
Status
Fibre B
Drive Pwr
Fibre B
Drive Pwr
Fibre A
Drive Pwr
Fibre B
Fibre A Status
Fibre B
Drive Pwr
Drive Pwr
Status
Fibre A
Status
Fibre A port to FCSW2
Fibre A
Status
Fibre A port to FCSW2
Fibre B
Drive Pwr
Fibre A port to FCSW2
Fibre B
Status
Fibre A port to FCSW2
Drive Pwr
Tape Library
Fibre A port to FCSW1
Fibre A port to FCSW1
Fibre A port to FCSW1
Fibre A port to FCSW1
23
Tiered Adaptive Storage™ Hardware Guide
IB Full TAS Configuration There are two types of full TAS configurations. The InfiniBand (IB) full TAS configuration consists of two management Ethernet switches, two FC switches, and two IB switches. The data cache block storage (DCBS) is a NetApp 5500. Figure 2-54 through Figure 2-67 show the FC full TAS configuration components and cabling. Figure 2-54 shows a second Ethernet switch; however, a second Ethernet switch is only used if
the system uses more than 2 DMs. The tape library cabling is determined by the site. Half the cables must go to FCSW1 and half must go to FCSW2. Figure 2-68 shows a possible tape library cabling configuration. Figure 2-54: ETHSW2 ETHSW2 VLAN1
VLAN2
esmaint-net 1
3
5
7
9
11
13
VLAN3
ipmi-net
15
17
19
21
23
25
27
VLAN4
site-admin-net metadata-net
29
31
33
35
37
39
41
43
LNK /ACT
45
FDX
DIAG
RPS
FAN
TEMP
PWR
47 /HDX COMBO PORTS
RESET LNK/ACT
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
LNK/ACT
LNK/ACT
LNK/ACT
48
Figure 2-55: ETHSW1 ETHSW1 VLAN1
VLAN2
esmaint-net 1
3
5
7
9
11
13
VLAN3
ipmi-net
15
17
19
21
23
25
27
VLAN4
site-admin-net metadata-net
29
31
33
35
37
39
41
43
45
LNK /ACT
FDX
DIAG
RPS
FAN
TEMP
PWR
47 /HDX COMBO PORTS
RESET LNK/ACT
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
42
44
46
LNK/ACT
LNK/ACT
LNK/ACT
48
Figure 2-56: FCSW2 FCSW2 2-3>
0-1>
8-9>
6-7>
4-5>
0
SANbox 5800
20
22
21
23
10-11> >
MDC to FCSW - Tape
To Primary and Secondary MDC, Slot 2, Port 2 and Slot 7, Port 2
Tape Library to FCSW MDC/DM
Tape Library, Fibre B Port
Figure 2-57: FCSW1 FCSW1 2-3>
0-1>
8-9>
6-7>
4-5>
0
SANbox 5800
20
22
21
23
10-11> >
MDC to FCSW - Tape
To Primary and Secondary MDC, Slot 2, Port 1 and Slot 7, Port 1
Tape Library to FCSW MDC/DM
Tape Library, Fibre A Port
Figure 2-58: IBSW2 To Primary DM 1, Slot 3, Port 2
To Secondary MDC 2, Slot 3, Port 2
To DCBS Cntr A, Port 1
IBSW2 MGT
1
2
3
4
5
6
0 7
0-1> 8
9
10
11
2-3> 12
13
14
15
4-5>
16
17
18
19
20
21
6-7> 22
23
24
25
8-9> 26
27
28
29
20 30
31
32
33
34
22 35
36
SX6036 CONSOLE
PS1
To Primary MDC 1, Slot 3, Port 2 To Secondary DM 2, Slot 3, Port 2 PS2
UID
RST
24
SANbox 5800
21 10-11>
Cray Private
12-13>
14-15>
16-17>
23
18-19>
To DCBS Cntrl B, Port 1
HR85-8500-A
System Architecture [2]
Figure 2-59: IBSW2 To Primary DM 1, Slot 3, Port 1
To Secondary MDC 2, Slot 3, Port 1
To DCBS Cntr A, Port 2
IBSW1 SX6036
MGT 1
2
3
4
5
6
0-1>
0 7
9
8
10
2-3>
11
13
12
14
4-5>
15
16
17
18
19
21
20
6-7> 22
23
24
25
8-9> 26
27
20
29
28
30
31
32
33
34
22 35
CONSOLE
36
PS1
To Primary MDC 1, Slot 3, Port 1 To Secondary DM 2, Slot 3, Port 1 PS2
UID
RST
SANbox 5800
21 12-13>
10-11>
23
18-19>
16-17>
14-15>
To DCBS Cntrl B, Port 2
Figure 2-60: CIMS CIMS 1 1
750 W
2 1
2
3
750 W
4
{
iDRAC
eth3 failover-net to second CIMS node eth2 to VLAN 2 - ipmi-net
To KVM
eth1 to VLAN 3 - site-admin-net iDRAC to VLAN 2 - ipmi-net
eth0 to VLAN 1 - esmaint-net
Figure 2-61: Secondary MDC 2 Slot 1, port 2 to ABS Cntr B Port 2
10GbE to Customer Network
{
Slot 1, port 1 to ABS Cntr A Port 2 Slot 2, port 2 to FCSW2 - Tape Slot 2, port 1 to FCSW1 - Tape
Slot 7, port 2 to FCSW1 - Tape Slot 7, port 1 to FCSW2 - Tape
Secondary MDC 2 4
1
ST
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
eth3 failover-net to MDC1
iDRAC to VLAN 2 - ipmi-net
Slot 3, Port 1 to IBSW1
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net
Slot 3, Port 2 to IBSW2
eth0 to VLAN 1 - esmaint-net
Figure 2-62: Primary MDC 1 Slot 1, port 2 to ABS Cntr B Port 1
10GbE to Customer Network
{
Slot 1, port 1 to ABS Cntr A Port 1
4
1
ST
Slot 2, port 2 to FCSW2 - Tape Slot 2, port 1 to FCSW1 - Tape
Slot 7, port 2 to FCSW1 - Tape Slot 7, port 1 to FCSW2 - Tape
Secondary MDC 2
2
6
5
7 2
1 3 750 W
1
2
3
750 W
4
iDRAC
iDRAC to VLAN 2 - ipmi-net
Slot 3, Port 1 to IBSW1
HR85-8500-A
Slot 3, Port 2 to IBSW2
eth3 failover-net to MDC1 eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Cray Private
25
Tiered Adaptive Storage™ Hardware Guide
Figure 2-63: Secondary DM 2 10GbE to Customer Network
{
Secondary DM 2 4
1
ST
2
6
5
7 2
1
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
3 750 W
1
2
3
750 W
4
iDRAC
iDRAC to VLAN 2 - ipmi-net
Port 1
Port 2
Lnk
Lnk
Ch 1
Lnk
eth2 to VLAN 4 - metadata-net Lnk
Ch 2
Lnk
Act
Lnk
Act
IB Host Card
CH 2
ID/ Diag
CH 1
eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-64: Primary DM 1 10GbE to Customer Network
{
Primary DM 1 4
1
ST
2
6
5
7 2
1
Slot 3, Port 2 to IBSW2 Slot 3, Port 1 to IBSW1
3 750 W
1
2
3
750 W
4
iDRAC
iDRAC to VLAN 2 - ipmi-net
eth2 to VLAN 4 - metadata-net eth1 to VLAN 3 - site-admin-net eth0 to VLAN 1 - esmaint-net
Figure 2-65: KVM KVM 100-240V~, 0.5A, 50/60 Hz
1
3
5
7
2
4
6
8
LAN
Video Keyboard and Mouse
esmaint-net or site-admin-net
Maintenance dongle to connect KVM to another server
Figure 2-66: Administrative Block Storage (ABS) Administrative Block Storage Ch 1 Lnk
Ch 2
Lnk
Lnk
Port 1
Port 2
Lnk Lnk Port 1
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
DC
Lnk Lnk
Lnk
AC
SAS Cntrl A Ch2 to MDC2 SAS Cntrl A Ch1 to MDC1 Cntrl A Ethernet Port 1 to esmaint-net
26
Ch 1 Lnk
Cray Private
Ch 2
Lnk
Lnk
Port 1
Port 2
Lnk Lnk
Drive Expansion Lnk
Lnk Lnk
Lnk
ID/ Diag
Port 1
Port 2
Lnk
Lnk
Ch 3
Ch 4
Lnk Lnk
Ch 5
Ch 6
Lnk
Drive Expansion Lnk
DC
Lnk
Lnk
ID/ Diag Lnk Lnk
Lnk
AC
SAS Cntrl B Ch2 to MDC2 SAS Cntrl B Ch1 to MDC1 Cntrl B Ethernet Port 1 to esmaint-net
HR85-8500-A
System Architecture [2]
Figure 2-67: Data Cache Block Storage (DCBS) Cntr A, Port 1 to IBSW2 Cntr A, Port 2 to IBSW1
DCBS (IB) 2 1
DC
AC
PSU A
PSU A PSU B
PSU B
Lnk
Port 1
Port 2
Lnk
Lnk
ESM A
Lnk
Ch 1
Lnk
Ch 2
Lnk
ESM B
ID/ Diag
Ch 1
Lnk
Act
Lnk
Act
Act
Lnk
Act
ESM A
Lnk
Ch 2
Lnk
CH 2
IB Host Card
CH 1
ID/ Diag
ESM B
IB Host Card
ESM B
Lnk
CH 2
Port 2
CH 1
Port 1
ESM B ESM A
ESM A 2 1
DC
PSU B
AC
PSU B PSU A
PSU A
Cntr B, Port 2 to IBSW1 Cntr B, Port 1 to IBSW2
Figure 2-68: Tape Library
HR85-8500-A
Status
Fibre A
Fibre B
Fibre B
Fibre A
Fibre A
Cray Private
Drive Pwr
Status
Fibre A
Status
Fibre B
Drive Pwr
Fibre B
Drive Pwr
Fibre A
Drive Pwr
Drive Pwr
Status
Fibre A
Drive Pwr
Fibre B
Status
Fibre B
Status
Fibre A port to FCSW2
Fibre A
Status
Fibre A port to FCSW2
Fibre B
Fibre A
Drive Pwr
Fibre A port to FCSW2
Fibre B
Status
Fibre A port to FCSW2
Drive Pwr
Tape Library
Fibre A port to FCSW1
Fibre A port to FCSW1
Fibre A port to FCSW1
Fibre A port to FCSW1
27
Tiered Adaptive Storage™ Hardware Guide
Hardware Architecture Cray Integrated Management Server (CIMS) A Cray Integrated Management Services (CIMS) node is the centralized management node that manages slave node power, software provisioning, reporting, and health status. The CIMS runs the Bright software and provides a centralized platform to manage the system hardware and software. Nodes that are managed by the CIMS are called slave nodes. The TAS CIMS node is a Dell PowerEdge R620 rack server. Figure 2-69: TAS Architecture - CIMS to Ethernet Cray Integrated Maintence Sevices (CIMS) node(s)
Port B
Port A
eth3
ipmi0 DRAC
eth2 eth3 NIC 1 NIC 1
CIMS node connections e eth0
eth2
eth1
DRAC ipmi0
failover-net
site-admin-net
failover-net fa
ipmi-net site-admin-net esmaint-net
Bright GUI
10.50.x.x
site-admin-net
site-admin-net
site-admin-net
ipmi-net
esmaint-net
10.149.x.x IB switch subnet mgt
eth1 eth0 NIC 1 NIC 1
metadata-net VLAN 4
site-admin-net VLAN 3
ipmi-net VLAN 2
esmaint-net VLAN 1
Ethernet switch(es) or VLANs
Metadata Controller (MDC) and Data Mover (DM) Nodes In each TAS system, the Dell R720's function as metadata controllers. These systems provide management of the VSM cache file system as well as managing the VSM policy engine. They also are responsible for the data movement between the cache file system and the tape archive media. Additional R720's function as data movers. These systems function as the ingress and egress of data into the TAS system. They may have file serving applications, HSM data mover tool, or other copy tool applications running on them. Refer to Figure 2-70 and Figure 2-71 for the MDC and DM architecture diagrams. The Ethernet labeling may differ from what is shown in the figures.
28
Cray Private
HR85-8500-A
System Architecture [2]
Figure 2-70: TAS Architecture - Ethernet to MDC and DM Nodes
metadata-net VLAN 4
site-admin-net VLAN 3
ipmi-net VLAN 2
esmaint-net VLAN 1
Ethernet switch(es) or VLANs Customer network 10GbE
metadata-net
10.141 10.148
customer-net metadata-net
site-admin-net ipmi-net esmaint-net 10.50
eth3 eth0
failover-net
DRAC
eth1 eth2 eth5 eth4 10GbE Primary VSM MDC FC8 IB/FC SAS
hte eth4 eth3 eth0 DRAC eth2 eth1 5eth5 10GbE Secondary VSM MDC Node SAS IB/FC FC8
eth0 DRAC eth1 eth2 eth4 eth5 TAS Client 10GbE (Data Mover Node) IB/FC
eth0 DRAC eth1 eth2 eth4 eth5 TAS Client 10GbE (Data Mover Node) IB/FC Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
esmaint-net network to IB and FC switches, RAID and tape controllers, and other devices managed by Bright 10.141.0.x Server Nodes 10.141.50.x Ethernet Switches 10.141.100.x RAID controllers 10.141.150.x FC/SAS switches 10.141.200.x IB switches
Figure 2-71: TAS Architecture - MDC/DM Node Connections 1GbE ipmi0 DRAC
eth1 eth0 NIC 1 NIC 1
eth2 NIC 1
10GbE eth3 NIC 1
eth5 eth4 NIC 2 NIC 2
TAS MDC/DM node connections ipmi-net
failover-net (MDC only) metadata-net site-admin-net esmaint-net
customer-net
Administrative Block Storage (ABS) TBD Refer to Figure 2-72 for the ABS architecture diagram. The Ethernet labeling may differ from what is shown in the figure.
Data Cache Block Storage (DCBS) TBD Refer to Figure 2-72 for the DCBS architecture diagram. The Ethernet labeling may differ from what is shown in the figure.
HR85-8500-A
Cray Private
29
Tiered Adaptive Storage™ Hardware Guide
Data Archive Storage Tape (DAST) TBD Refer to Figure 2-72 for the DAST architecture diagram. The Ethernet labeling may differ from what is shown in the figure. Figure 2-72: TAS Architecture - MDC/DM Nodes to Switches, DCBS, and DAST
eth3 eth0
eth3 eth0 DRAC eth2 eth1 5eth5 hte eth4 10GbE Secondary VSM MDC Node SAS IB/FC FC8
DRAC
eth1 eth2 eth5 eth4 10GbE Primary VSM MDC IB/FC SAS FC8
Port A
Cntrl B
Cntrl A
To DCBS via IB switch
eth0 DRAC eth1 eth2 eth4 eth5 TAS Client 10GbE (Data Mover Node) FC Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port A
Port B
Port B
Port A
Port A
Port B
Port A
Port B
Disks or SSDs
eth0 DRAC eth1 eth2 eth4 eth5 TAS Client 10GbE (Data Mover Node) FC
To DCBS via IB switch
Administrative Storage (MABs) 10.149 IB Switch
IB Switch
Cntrl B
Cntrl A
Cntrl B
Cntrl A
LUNs Data cache block storage (DCBS)
LUNs Data Archive Storage Tier (DAST)
FC8 Switch FC8 Switch S
Tape Archive S Storage Tier (TAST)
30
Cray Private
HR85-8500-A
Site Preparation [3] Site Planning Proper site planning and preparation is important for the successful installation of the TAS (tiered adaptive storage) system; Cray site planning personnel (
[email protected]) will assist with the site planning process. Each site has different site planning characteristics to consider. You can guarantee customer’s effective site planning by identifying their initial system configuration as well as any upgrade plans. Qualified electrical and mechanical facility engineers should be involved early in the site planning process. Before customers can begin site preparation activities, they must prepare electrical and mechanical design drawings for approval by Cray site planning personnel. The following subsections describe various considerations and requirements that the site planning and preparation process includes.
System Specifications Refer to Table 3-2 for the TAS system rack specifications and to Figure 3-1 for the dimensions and clearance requirements. Table 3-2: TAS System Specifications Characteristic
HR85-8500-A
Specification
Height
78.40 in. (1991 mm)
Width
23.60 in. (600 mm)
Depth
47.25 in. (1200 mm)
Weight (maximum)
Varies
Shipping weight (maximum)
Varies
Cray Private
31
Tiered Adaptive Storage™ Hardware Guide
Table 3-2: TAS System Specifications Access Requirements Front Rear Side Acoustical noise level
Varies
Heat dissipation to air (maximum)
Varies
Cooling requirement
Air cooled
Air flow
Intake: front Exhaust: rear
Input voltage Nominal Tolerance
32
36.00 in. (914 mm) 36.00 in. (914 mm) none
North American/Japan 200/208Vac 185-264Vac
Frequency
60 or 50 Hz
Phases
1 phase
Power requirement (maximum)
7.89 kVA (7.50 kW)
Hold-up time
8 milliseconds
Power circuit (4 required)
North American/Japan Hubbell L6-30R
Power circuit size (4 required)
30 amp at 200/208Vac 32 amp at 230Vac
Power cord length
8 ft (2.4 m)
Cray Private
International 230Vac 185-264Vac
International Hubbell 332C6W
HR85-8500-A
Site Preparation [3]
Figure 3-1: TAS System Cabinet Dimensions Plan view 11.80 (300 mm)
23.60 (600 mm)
Front view
Side view
23.60 (600 mm)
47.25 (1200 mm)
78.40 (1991 mm)
Shipping Configurations Figure shows the domestic and international TAS cabinet shipping configurations. Each TAS
cabinet is shipped on a separate shipping pallet. Cray provides an attached ramp to facilitate the removal of the cabinet from the shipping pallet. You must provide a pallet jack to move each shipping pallet to the system location. International shipments include a wooden crate.
HR85-8500-A
Cray Private
33
Tiered Adaptive Storage™ Hardware Guide
Figure 3-2: TAS Shipping Configurations
U.S. domestic shipping configuration
International shipping configuration
Site Access Requirements Before system installation, the customer site must meet certain site access requirements. This subsection explains these requirements and provides specifications. For computer system delivery, the loading dock should be approximately 46 in. to 50 in. (1168 mm to 1270 mm) high. Take special precautions when moving equipment if the loading dock or access route has an engraved floor pattern; an engraved pattern can cause vibration damage to computer equipment on casters. The equipment access route should have no slope greater than 10 degrees. If no loading dock exists or if the loading dock does not meet the requirements, the customer must provide a forklift to unload the computer equipment. Refer to Forklift Size Requirements for the Handling of Cray Inc. Equipment, Site Engineering document 10658374, for forklift requirements. If the computer room is on a different level than the loading dock and the building does not have an elevator, the customer may need a crane or other special handling equipment to lift the computer equipment to the same level as the computer room.
34
Cray Private
HR85-8500-A
Site Preparation [3]
The access route must be maintained for initial equipment delivery and for parts delivery during equipment maintenance. The entire access route from the loading dock to the computer room must meet the requirements in Table 3-3. Table 3-3: Access Route Requirements Shipping Specifications Cabinet
Height
Width
Depth
Weight
Tiered Adaptive Storage (TAS)
78.40 in. (1991 mm)
23.60 in. (600 mm)
47.24 in. (1200 mm)
Varies
Computer Room Environment Cray designs computer equipment to operate within specific ranges of air quality, temperature, and relative humidity levels. Significant variations of these levels in a computer room environment can cause disruptions in equipment operation and decrease the life of the equipment. The TAS system must operate in a controlled computer room environment. The following requirements refer to the entire computer room, as they also affect air-cooled devices such as disk drives and servers. Therefore, the design and layout of the environmental control equipment (such as computer room air-conditioning units) must ensure that inlet air to the aircooled device meets these environmental requirements.
HR85-8500-A
Cray Private
35
Tiered Adaptive Storage™ Hardware Guide
The computer system requires a computer room environment that is monitored and remains within the parameters in Table 3-4. Table 3-4: TAS Air Quality, Temperature, and Relative Humidity Levels Temperature (Ambient Air) Operating1 Nonoperating
60°F to 80°F (16°C to 27°C) 32°F to 140°F (0°C to 60°C)
Temperature (Underfloor Air) Range
60°F to 80°F (16°C to 27°C)
Relative Humidity (Ambient Air) Operating* Nonoperating
30% to 65% noncondensing 5% to 80% noncondensing
Relative Humidity (Underfloor Air) Operating
30% to 90% noncondensing
Altitude Operating Nonoperating
0 to 10,000 ft (0 to 3,048 m) 0 to 40,000 ft (0 to 12,192 m)
1. Cray recommends an operating temperature range of 70°F to 77°F (21°C to 25°C) and a relative humidity of 30% to 50% noncondensing.
36
Cray Private
HR85-8500-A
Site Preparation [3]
Electrical Requirements Cray makes every effort to minimize the effects of power failures and interruptions to the hardware. However, if the computer system experiences repeated power interruptions and fluctuations, it may also experience a higher component failure rate than with a stable power source. If your input power is unstable, Cray encourages you to install a reliable power source, such as a UPS, to reduce the possibility of component failures. The TAS system requires one of the following types of power: North America/Japan: • 200/208Vac, 1 phase, 50 or 60Hz International: • 230Vac, 1 phase, 50 or 60Hz Table 3-5 provides additional electrical service requirements. Table 3-5: Electrical Service Requirements Electrical Service
Requirement
Phase imbalance
5% maximum (line-to-line, line-to-line neutral)
Voltage harmonics
5% maximum total, 3% largest
Voltage deviation from sine wave
5% to -10%
Voltage modulation
3% maximum
Transient voltage surges
+5%
Transient voltage sags
-5%
Frequency tolerance
5%
Frequency rate of change
Less than 1.0 Hz during any 10-cycle period
Total kilowatt power requirements depend on system configuration and equipment upgrade plans. Cray will provide documentation that customers can use to estimate the power requirements for their specific system configuration by contacting Site Engineering at
[email protected].
Equipment Grounding All Cray computer equipment requires a protective power safety-ground system. The power safety-ground system protects personnel from shock hazards and protects the computer equipment from damage caused by electrical malfunctions. Local and national electrical codes regulate the power safety-ground system.
HR85-8500-A
Cray Private
37
Tiered Adaptive Storage™ Hardware Guide
All Cray computer equipment also requires a signal reference grid. The signal reference grid establishes an equipotential reference plane for high-frequency digital signals between interconnected computer equipment. Cray supplies braided ground straps with all equipment. Customers are responsible for connecting the ground straps to the signal reference grid. Customers can obtain Equipment Grounding for Cray Inc. Computer Systems, document 10658002, by contacting Site Engineering at
[email protected]. This document describes the grounding system requirements and identifies alternative methods for providing the signal reference grid. In addition, the document describes electrostatic discharge (ESD) precautions and maintenance of the facility’s grounding systems. Customers must provide, install, and maintain the approved grounding systems as described in the equipment grounding document and in this subsection.
Floor Requirements Refer to Figure 3-3 for the location of the floor cutouts for the TAS system. Each TAS system requires a floor cutout for the entry of data and power cables. Cray recommends that you place a perforated floor panel in front of each RAID cabinet to provide additional cooling air. Figure 3-3: TAS System Floor Cutout
38
Cray Private
HR85-8500-A
Hardware Implementation [4] Rack and PDUs TAS System Rack The TAS system is stored in an APC 42U rack. More information on the rack can be found in Chapter 3, "Site Preparation," on page 31.
Power Distribution Units (PDUs) Four 220VAC power distribution units (PDUs) are mounted to the rear of the rack. Each PDU can be supplied with power from a different source for power redundancy. A circuit breaker at the bottom of each PDU applies power to the components in the rack.
Cray Integrated Management Server (CIMS) Marketing Code: XC-TAS01-CIMS-01 A Cray Integrated Management Services (CIMS) node is the centralized management node that manages slave node power, software provisioning, reporting, and health status. The CIMS runs the Bright software and provides a centralized platform to manage the system hardware and software. Nodes that are managed by the CIMS are called slave nodes. The TAS CIMS node is a Dell PowerEdge R620 rack server.
Specifications The Cray TAS CIMS node includes a 4 hard drive bay.
HR85-8500-A
•
Dell R620 - 2.5in chassis with up to 8 hard drives
•
Two Intel Xeon E5-2609v2 2.5 GHz, 10 MB cache, 6.4 GT/s QPI, No Turbo, 4-core, 80 Watts, 1333 MHz max memory
•
Eight 8-GB RDIMM, 1333 MHz RDIMMs (64 GB total memory)
•
One PERC H310 RAID controller
Cray Private
39
Tiered Adaptive Storage™ Hardware Guide
•
Four 1-TB 7.2K RPM SATA 3 Gbps 2.5" hard drives
•
RAID 5 3+1
•
One DVD ROM SATA drive
•
ReadyRails sliding rails without cable management arm
•
Two 495 Watt (1+1) power supplies
•
iDRAC 7 Enterprise
Figure 4-1: System Board Block Diagram
40
Cray Private
HR85-8500-A
Hardware Implementation [4]
CIMS Controls and Indicators Figure 4-2: CIMS Front Panel Features and Indicators USB connector (2) System identification button NMI button
vFlash media card slot LCD panel
Optical drive (optional)
LCD menu buttons
Information tag Video connector
Power-on Indicator, power button
Hard drives (8)
Table 4-1: CIMS Front Panel Features and Indicators Control/Indicator
Description
Power-on indicator
Lights when the system power is on
NMI button
Used to troubleshoot software and device driver errors when running certain operating systems
System identification button
Used to locate a particular system within a rack
USB connector (2) Optical drive (optional)
SATA DVD-ROM drive or DVD+/-RW drive
vFlash media card slot LCD menu buttons
Used to navigate the LCD menu
LCD panel
Displays system ID, status information, and system error messages. The LCD lights blue during normal system operation
Information tag
Records the service tag, NIC, and MAC address
VGA video connector Hard drives (8)
HR85-8500-A
Supports up to four 2.5 hot-swappable hard drives
Cray Private
41
Tiered Adaptive Storage™ Hardware Guide
Figure 4-3: CIMS Rear Panel Features and Indicators,
System identification button
System identification connector
iDRAC7 enterprise port
Serial connector
Ethernet connectors (4)
VGA video connector PCIe expansion card slot (riser 2)
PCIe expansion card slot (riser 3)
Power supply (PSU1)
Power supply (PSU2)
USB connectors (2)
Table 4-2: CIMS Rear Panel Features and Indicators Control/Indicator System identification button
Description Used to locate a particular system within a rack
System identification connector iDRAC7 enterprise port Serial connector PCIe expansion card slot (riser 2) VGA video connector USB connectors (2) Ethernet connectors (4)
Four Ethernet or NIC connectors
PCIe expansion card slot (riser 3)
42
Power supply (PSU1)
Learn what AC/DC power we use
Power supply (PSU2)
Learn what AC/DC power we use
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-4: CIMS Components
CIMS IO Configuration Network interface controller 1 (NIC), which is built into the motherboard, provides Ethernet connections for the TAS networks. CIMS nodes are configured with the following networks. The Ethernet labeling may differ from what is stated: •
esmaint-net Private network (eth0) for system administration and maintenance. This network is used to provision slave nodes with software images and is used by Bright Cluster Management software.
•
ipmi-net Private maintenance network (eth2) for power control and monitoring.
•
site-admin-net System administration network (eth1 and impi0).
•
HR85-8500-A
The iDRAC port (management controller) is used for system administration and remote power control.
Cray Private
43
Tiered Adaptive Storage™ Hardware Guide
Metadata Controller (MDC) and Data Mover (DM) Nodes Marketing code: XC-TAS01-CMCS-01 The metadata controller (MDC) and data mover (DM) nodes are built from generic Lustre File System by Cray (CLFS) nodes. The metadata controller nodes are redundant and are configured for high-availability. DM nodes move data from customer file systems to the TAS system.
Specifications The MDC and DM nodes contain the following hardware components. Figure 4-5 shows the system board block diagram for the nodes:
44
•
Two (2) Intel Xeon E5-2650v2 2.6 GHz, 20 MB cache, 8.0 GT/s QPI, Turbo, HT, 8-core, 95 Watts, 1866 MHz max memory
•
MDC: 16 8-GB RDIMM, 1866 MHz RDIMMs (128 GB total memory)
•
DM: 8 8-GB RDIMM, 1866 MHz RDIMMs (64 GB total memory)
•
One (1) PERC H710 RAID controller 512 GB cache
•
Two (2) 600-GB 10K RPM SAS 6 Gbps 2.5" hard drives
•
RAID 1
•
One (1) DVD ROM SATA drive
•
ReadyRails sliding rails without cable management arm
•
Two (2) 750 Watt (1+1) power supplies
•
iDRAC 7 Enterprise
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-5: MDC and DM Node System Block Diagram
USB2.0 Key
RJ45
DELL Se lect Networ k Adapte r Front Video Rear Video
Two Rear USB2.0 Ports
USB2.0
IDSDM
One Front USB2.0 Ports
Serial Port
To Front Panel 2x30
DIMM3
DIMM3
DIMM3
DIMM3
DIMM3
DIMM3
DIMM3
DIMM3
DIMM2
DIMM2
DIMM2
DIMM2
DIMM2
DIMM2
DIMM2
DIMM2
DIMM1
DIMM1
DIMM1
DIMM1
DIMM1
DIMM1
DIMM1
DIMM1
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
CH2
CH3
CH0
CH1
CH2
CH3
TPM
Ie
Ge n2
QPI0
SAS A/B (To Backplane)
(Port 2C, 2D) B) x8 ,1 1A ort (P
x8
x2
, 2A ort (P
2B
x8
DELL Select Network Adapter Slot
Center Riser (Slot 5)
)
) B) 3D ,3 C, 3A rt 3 ort (P (Po
x8
x8
QPI 1.1
QPI0
QPI 1.1
QPI1
CPU2 rt 1 (Po
A,
) 1B
Storage Connector
X16 Left Riser (Slot 6)
ort
, 2A
2B
) (P
x8
x8
x8
Storage Connector
Rear External SD
(P
Right Riser (Slot 1)
ort
Center Riser PCIe x16 Connector (Slot 4)
2C
,2
D) ort (P
x8
, 3A
) 3B
(Po
C, rt 3
) 3D
x8 PCIe Gen3 x8
PC
CPU1
PCIe Gen3 x8
PCIe x1 port 3 + port 4
Port 0(DMI)
PCIe Gen3 x8
QPI1 DMI 2GB/s
PCIe Gen3 x8
TPM
South Bridge
PCIe Gen3 x8
PCIe
LPC
PCIe Gen3 x8
SD
LPC
PCIe Gen3 x8
LPC
PCIe Gen3 x8
PCIe
MAC
CH1
PCIe Gen3 x8
PCIe
CH0 MiniSAS Conn
USB
PCIe Gen3 x8
iDRAC
Coin Cell
SAS x8
USB
SATA
USB
MGMT MAC
LPC
Video UART SPI
Right Riser (Slot 2) & (Slot 3)
Internal PERC
System Power Connectors
BP Power Conns. 4X
Power Supply 1
Power Supply 2
(495W /750W/ 1100W )
(495W /750W/ 1100W )
HR85-8500-A
GPGPU Power Conn.
BP Signal Conns. 4X
Cray Private
45
Tiered Adaptive Storage™ Hardware Guide
MDC and DM Controls and Indicators Figure 4-6: MDC and DM Front Panel Features and Indicators
Table 4-3: MDC and DM Front Panel Features and Indicators Control/Indicator
Description
Power-on indicator
Lights when the system power is on
NMI button
Used to troubleshoot software and device driver errors when running certain operating systems
System identification button
Used to locate a particular system within a rack
VGA video connector LCD menu buttons
Used to navigate the LCD menu
Information tag
Records the service tag, NIC, and MAC address
LCD panel
Displays system ID, status information, and system error messages. The LCD lights blue during normal system operation
Optical drive (optional)
SATA DVD-ROM drive or DVD+/-RW drive
Hard drives
Supports eight 2.5 hot-swappable hard drives
vFlash media card slot USB connector (2) Tape drive slot (optional)
46
Supports one optional 3.5 inch tape backup unit
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-7: MDC and DM Rear Panel Features and Indicators
Table 4-4: MDC and DM Rear Panel Features and Indicators Control/Indicator
Description
System identification button
Used to locate a particular system within a rack
System identification connector iDRAC7 enterprise port PCIe expansion card slots low-profile (3) Serial connector VGA video connectors (2) USB connectors (2) Ethernet connectors (4)
Four Ethernet or NIC connectors
PCIe expansion card slots full height (4)
HR85-8500-A
Power supply (PSU1)
Learn what AC/DC power we use
Power supply (PSU2)
Learn what AC/DC power we use
Cray Private
47
Tiered Adaptive Storage™ Hardware Guide
Figure 4-8: MDC and DM Inside the System
MDC and DM IO Configuration Network interface controller 1 (NIC), which is built into the motherboard, provides Ethernet connections for the TAS networks. In addition to NIC1, a second NIC (10GbE) provides network connections for the customer network. MDC and DM nodes are configured with the following networks. The Ethernet labeling may differ from what is stated: •
esmaint-net Private network (eth0) for system administration and maintenance. This network is used to provision slave nodes with software images and is used by Bright Cluster Management software.
•
ipmi-net iDRAC private maintenance network (impi0) for power control and monitoring.
•
site-admin-net System administration network (eth1).
•
metadata-net Metadata network (eth2) for storage management software.
48
Cray Private
HR85-8500-A
Hardware Implementation [4]
CIMS, MDC, and DM Nodes The following indicators are identical for the CIMS, MDC, and DM nodes. Figure 4-9: CIMS, MDC, and DM LCD Panel Features
The LCD panel provides status and error messages. •
The LCD backlight lights blue during normal operating conditions and lights amber to indicate an error condition.
•
The LCD backlight is off when the system is in standby mode and can be turned on by pressing either the Select, Left, or Right button on the LCD panels.
•
The LCD backlight remains off if LCD messaging is turned off through the iDRAC utility, the LCD panel, or another tools.
1.
The left button moves the cursor back in one-step increments.
2.
The select button selects the menu item highlighted by the cursor.
3.
The right button moves the cursor forward in one-step increments. During message scrolling: • Press once to increase scrolling speed • Press again to stop • Press again to return to default scrolling speed • Press again to repeat the cycle
HR85-8500-A
Cray Private
49
Tiered Adaptive Storage™ Hardware Guide
Figure 4-10: CIMS, MDC, and DM Hard Drive Indicators
Table 4-5: CMS, MDC, and DM Drive-Status Indicator Patterns Drive-Status Indicator Patterns (RAID Only)
Condition
Blinks green two times per second
Identifying drive or preparing for removal
Off
Drive ready for insertion or removal
Blinks green, amber, and off
Predicted drive failure
Blinks amber four times per second
Drive failed
Blinks green slowly
Drive rebuilding
Steady green
Drive online
Blinks green three seconds, amber three seconds, and off six seconds
Rebuild aborted.
Each AC power supply has an illuminated translucent handle and each DC power supply (when available) has an LED that serves as an indicator to show whether power is present or whether a power fault has occurred. Figure 4-11: CIMS, MDC, and DM AC Power Supply Status Indicators
50
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-12: CIMS, MDC, and DM DC Power Supply Status Indicators
Table 4-6: CIMS, MDC, and DM Power Indicator Pattern Power indicator pattern
HR85-8500-A
Condition
Not lit
Power is not connected
Green
The LED indicator lights green indicating that a valid power source is connected to the power supply and that the power supply is operational.
Flashing amber
Indicates a problem with the power supply or power supply mismatch
Flashing green
Power supply is mismatched
Cray Private
51
Tiered Adaptive Storage™ Hardware Guide
Keyboard, Video, and Mouse (KVM) Figure 4-13: Rear View of an 8-port Rack Mount LCD KVM Assembly KVM 100-240V~, 0.5A, 50/60 Hz
1
3
5
7
2
4
6
8
LAN
Video esmaint-net or site-admin-net
Keyboard and Mouse
Maintenance dongle to connect KVM to another server
Table 4-7: KVM Monitor Push-button Functions Button/LED
Description
AUTO EXIT
Exits from the OSD function or returns to previous menu
SL (+ or -)
Moves the selector right or left on the OSD menu
MENU SELECT
Enables the OSD menu and controls the function to be adjusted
(Insert Power Symbol)
Turns your rack mount LCD KVM assembly on or off
LED (status light)
Green = on Flashing green = power save mode or disconnection of signal cable Red = off
Hotkey Functions The 8-port version of the Rack Mount LCD KVM Assembly features hotkey commands that allow you to easily access various functions of your assembly. The following table explains the hotkey functions available. Note: To utilize the hotkey functions in the following table, press the Ctrl key on the left side of the keyboard twice, then press the keys for the desired command. Table 4-8: KVM Hotkey Commands and Descriptions Function
Keystrokes
Description
Left cntl + left ctrl + 1-8
Selects the desired port channel by number (port 1-8)
Left cntl + left ctrl + F1-F8
Selects port channels 9-16
Left ctrl + left cntl + up or down arrow
The up arrow selects the next channel; the down arrow selects the previous channel
Beep
Left cntl + left ctrl + b
Turns the beep function on or off in scan mode (default is on)
Auto scan
Left ctrl + left ctrl + s
Allows you to exit Auto Scan by pressing any key
Reset assembly
Left ctrl + left ctrl + r
Resets the LCD KVM assembly
Port switch
52
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-8: KVM Hotkey Commands and Descriptions (continued) Function
Keystrokes
Description
Reset mouse
Left ctrl + left ctrl + m
Resets the mouse
Reset keyboard
Left ctrl + left ctrl + k
Reset the keyboard
OSD menu
Left ctrl + left ctrl + spacebar (doubleclick the right mouse button)
Displays the OSD menu
The two consecutive left ctrl keystrokes and the following command key(s) should be pressed within two seconds or the hotkey sequence will not be validated. Port number selections must be made using the numeric keys on the keyboard. Numeric keys on the keypad are not available as hotkey commands.
HR85-8500-A
Cray Private
53
Tiered Adaptive Storage™ Hardware Guide
TAS Metadata Block Storage Devices Introduction Marketing Code: NG6-DE60-C2708-4FC16 The NetApp 2700-24 is known as the Administrative Block Storage (ABS). It is used for cache file system metadata, and also an administrative file system (name = tas_admin). The admin data consists of data including logs and reports, and config files that have to be kept in sync across the two metadata controllers. The NetApp 5400-60 and 5500-60 are used as Data Cache Block Storage (DCBS) devices. The NetApp 5400-60 supports Fibre Channel and the NetApp 5500-60 supports InfiniBand. A NetApp 2700 with 2 IB RAID controllers could be used as a small metadata block storage (MDBS) instead of a NetApp 5500-60 storage device.
Controller Types Administrative Block Storage (ABS) SAS Interface, Controls, and Indicators •
The ABS (administrative block storage) is a NetApp E2700-24 storage device with metadata solid state disk drives. The admin LUN is four 600 GB drives, RAID 5 3+1.
•
The configuration of the host ports might appear different on your system depending on which host interface card configuration is installed. The controls and indicators might appear in different locations depending on which model is used for the CIMS, MDS, etc.
Figure 4-14: ABS Controller-Drive Tray - Front View End cap locate LED
Drive canister
End cap service action required LED End cap over-temperature LED End cap power LED End cap standby power LED
54
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-15: ABS Controller-Drive Tray - Rear View SAS host 1 link up LED Serial port USB port
SAS host 1 link fault LED
SAS host 2 link up LED
1GbE ethernet management port 2
Tray ID indicator display
SAS host 2 link fault LED SAS expansion link 1 fault LED
Ethernet port 2 link active led
Controller canister
Power-fan canister
SFF-8644 SAS host connector 1 SFF-8466 SAS host connector 2
Ethernet port 1 link rate LED
AC power connector and switch
SAS expansion link 1 up LED
SFF-8644 SAS connector 1 (expansion)
Cache active led Controller service action required LED
DC power connector and switch
Controller service action allowed LED Battery charging LED Battery Service Action Required LED
Figure 4-16: ABS Right-Rear Subplate with No Host Interface Card Expansion SFF-8088 port connector ESM expansion active LED ESM expansion fault LED
Lnk
Port 1 Port 2
Lnk
Lnk
Lnk
Drive Expansion S A S
HR85-8500-A
Cray Private
55
Tiered Adaptive Storage™ Hardware Guide
Figure 4-17: ABS Right-Rear Subplate with a Two-Port 12-Gb/s SAS Host Interface Card 3. SFF-8644 SAS port 1 (expansion) SAS expansion link 1 up LED SAS expansion link 1 fault LED
Lnk
Port 1 Port 2
Lnk
Lnk Lnk
Ch 3
Ch 4
Lnk
Lnk
S A S
Drive Expansion Lnk
Lnk
S A S
ABS Host Channels
Caution: To prevent ESD damage to the tray, use proper ESD protection when you handle tray components.
•
Each controller has two native host ports.
•
In addition, you can have additional host ports a located on a host interface card (HIC). The following types of HICs are supported for TAS: •
Two SAS connectors at 6 Gb/s
Figure 4-18: ABS Host Channels on the Controller with a Two-Port SAS HIC - Rear View HD mini-SAS host interface card channel 4 HD mini-SAS host interface card channel 3
56
Cray Private
HR85-8500-A
Hardware Implementation [4]
LEDs on the ABS Controller-Drive Tray Figure 4-19: ABS LEDs on the Left End Cap
Controller-drive tray locate LED Service action required LED Controller-drive tray overtemperature LED Power LED
Standby Power LED
Table 4-9: ABS LEDs on the Left End Cap Location
HR85-8500-A
LED
Color
On
Off
1
Controllerdrive tray locate
White
Identifies a controller-drive tray that you are trying to find.
Normal status
2
Service action required
Amber
A component within the controller-drive tray needs attention.
Normal status
3
Controllerdrive tray overtemperature
Amber
The temperature of the controller-drive tray has reached an unsafe level
Normal status
4
Power
Green
Power is present
Power is not present
5
Standby power
Green
The controller-drive tray is in Standby Power mode.
The controller-drive tray is not in Standby Power mode.
Cray Private
57
Tiered Adaptive Storage™ Hardware Guide
Figure 4-20: ABS LEDs on the Drive
Drive power LED Drive service action required LED Drive service action allowed LED
Table 4-10: ABS LEDs on the Drive Location
LED
Color
On
Blinking Drive I/O activity is taking place.
Off The power is turned off.
1
Drive power
Green
The power is turned on, and the drive is operating normally.
2
Drive service action required
Amber
An error has occurred.
Normal status.
3
Drive service action required
Blue
The drive canister can be removed safely from the controller-drive tray.
The drive canister cannot be removed safely from the controller-drive tray.
Table 4-11: ABS Drive State LEDs
58
Drive State
Drive Power LED (Green)
Drive Service Action Required LED (Amber)
Power is not applied.
Off
Off
Normal operation - The power is turned on, but drive I/O activity is not occurring.
On
Off
Normal operation - Drive I/O activity is occurring
Blinking
Off
Service action required - A fault condition exists, and the drive is offline
On
On
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-21: ABS LEDs on the Controller Canister Main Faceplate Ethernet connector 1 link active LED
Host link 1 service action required LED
Ethernet connector 2 link rate LED
Host link 1 service action allowed LED
Ethernet connector 1 link active LED
Host link 2 service action required LED Host link 2 service action allowed LED
Ethernet connector 1 link rate LED
Lnk
Ch 1
S A S
Ch 2
Lnk
Lnk Port 1
Lnk
Port 2
ID/ Diag
Battery service action required LED
Tray ID indicator Cache active LED Controller service action required LED
Battery charging LED
Controller service action allowed LED
Table 4-12: ABS LEDs on the Controller Canister Main Faceplate Location
HR85-8500-A
LED
Color
On
Off
1
Ethernet connector 1 link rate LED
Green
There is a 1000 GB/s rate.
There is a 10BASE-T rate.
2
Ethernet connector 1 link active LED
Green
The link is up (LED blinks when there is activity).
The link is not active.
3
Ethernet connector 2 link rate LED
Green
There is a 1000 Gb/s rate.
There is a 10BASE-T rate.
4
Ethernet connector 2 link active LED
Green
The link is up (the LED blinks when there is activity).
The link is not active.
Cray Private
59
Tiered Adaptive Storage™ Hardware Guide
Table 4-12: ABS LEDs on the Controller Canister Main Faceplate Location
60
LED
Color
On
Off
5
Host link 1 service action required LED
Amber
At least one of the four PHYs is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
6
Host link 1 service action allowed LED
Green
At least one of the four PHYs in the host interface port is working and a link exists to the device connected to the IN port connector.
A link error has occurred.
7
Host link 2 service action required LED
Amber
At least one of the four PHYs is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
8
Host link 2 service action allowed LED
Green
At least one of the four PHYs in the host interface port is working and a link exists to the device connected to the IN port connector.
A link error has occurred.
9
Battery service action required LED
Amber
The battery in the controller canister has failed.
Normal status.
10
Battery charging LED
Green
The battery is fully charged. The LED blinks when the battery is charging.
The controller canister is operating without a battery or the existing battery has failed.
11
Controller service action allowed LED
Blue
The controller canister can be removed safely from the controllerdrive tray.
The controller canister cannot be removed safely from the controller-drive tray.
12
Controller service action required LED
Amber
A fault exists within the controller canister.
Normal status.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-12: ABS LEDs on the Controller Canister Main Faceplate Location
LED
Color
On
Off
13
Cache active LED
Green
Cache is active*
Cache is inactive or the controller canister has been removed from the controllerdrive tray.
14
Tray ID indicator display
Green
For more information, refer to “Controller-Drive Tray Sequence Code Definitions” on page 114.
* After an AC power failure, this LED blinks while cache offload is in process.
Figure 4-22: ABS LEDs on the Controller Canister Two-Port SAS Host Interface Card Subplates* Host interface card link 4 up LED
Host interface card link 5 up LED
Host interface card link 4 active LED
Host interface card link 5 active LED
Host interface card link 3 active LED
Host interface card link 6 up LED
Host interface card link 3 up LED
Host interface card link 6 active LED Lnk
S A S Lnk
Ch 3
Ch 4
Port 2
Lnk Lnk
Lnk
Ch 5
Lnk
Lnk
Lnk Lnk
Ch 6 Lnk
Lnk
Port 1
Lnk
Drive Expansion 12Gb/s
S A S
Table 4-13: ABS LEDs on the Controller Canister Two-Port SAS Host Interface Card Subplates*
HR85-8500-A
Location
LED
Color
On
Off
1
Host interface card link 3 up LED
Green
The Ethernet link has auto-negotiated to 1 GB/s.
The Ethernet link is down or does not auto-negotiate to 1 GB/s.
2
Host interface card link 3 active LED
Green
The link is up (LED blinks when there is activity).
The link is not active.
3
Host interface card link 4 up LED
Green
The Ethernet link has auto-negotiated to 1 GB/s.
The Ethernet link is down or does not auto-negotiate to 1 GB/s.
4
Host interface card link 4 active LED
Green
The link is up (LED blinks when there is activity).
The link is not active.
5
Host interface card link 5 up LED
Green
The Ethernet link has auto-negotiated to 1 GB/s.
The Ethernet link is down or does not auto-negotiate to 1 GB/s.
Cray Private
61
Tiered Adaptive Storage™ Hardware Guide
Table 4-13: ABS LEDs on the Controller Canister Two-Port SAS Host Interface Card Subplates* Location
LED
Color
On
Off
6
Host interface card link 5 active LED
Green
The link is up (LED blinks when there is activity).
The link is not active.
7
Host interface card link 6 up LED
Green
The Ethernet link has auto-negotiated to 1 Gb/s.
The Ethernet link is down or does not auto-negotiate to 1 GB/s.
8
Host interface card link 6 active LED
Green
The link is up (LED blinks when there is activity).
The link is not active.
* The figure shows a 4-port SAS host interface card subplate
Figure 4-23: ABS LEDs on the Power-Fan Canister
Standby power LED
Power-fan AC power LED
Power-fan DC power LED
Power-fan service action required LED
Power-fan service action allowed LED
Table 4-14: ABS LEDs on the Power-Fan Canister
62
Location
LED
Color
On
Off
1
Standby power
Green
The controller-drive tray is in Standby mode, and Dc power is not available.
The controller-drive tray is not in Standby mode, and DC power is available.
2
Power-fan DC power
Green
DC power from the powerfan canister is available.
DC power from the powerfan canister is not available.
3
Power-fan service action allowed
Blue
The power-fan canister can be removed safely from the controller-drive tray.
The power-fan canister cannot be removed safely from the controller-drive tray.
4
Power-fan service action required
Amber
A fault exists within the power-fan canister.
Normal status.
5
Power-fan AC power
Green
AC power to the power-fan canister is present.
AC power to the power-fan canister is not present.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Data Cache Block Storage (DCBS) FC Interface, Controls, and Indicators Marketing Code: NG6-DE60-C5512-8FC16 •
The data cache block storage (DCBS) FC is a NetApp 5400-60 with optional expansion trays.
•
The configuration of the host ports might appear different on your system depending on which host interface card configuration is installed. The controls and indicators might appear in different locations depending on which model is used for the CIMS, MDCs, etc.
Figure 4-24: DCBS (FC) Controller-Drive Tray - Front View
Drive drawer Locate LED Service action required LED Overtemperature LED Power LED Standby power LED
Figure 4-25: DCBS (FC) Controller-Drive Tray - Rear View Ethernet link 2 active LED Ethernet link 1 rate LED Ethernet connector 1
Fibre channel link 1 up LED
Fibre channel link 1 active LED
Ethernet link 2 rate LED Ethernet connector 2
Fibre channel link 2 up LED
Fibre channel connector 1
Fibre channel connector 2
Mini USB connector
Fibre channel link 2 active LED
Expansion fault LED
Fibre channel connector 4
Ethernet link 1 active LED
Expansion active LED Expansion SFF-8088 port connector Controller A
USB connector Fan canister Fan power LED Fan service action required LED Fan service action allowed LED
Battery service action required LED
Second tray ID indicator display field
Battery charging LED Controller A service action allowed LED Controller A service action required LED Cache active LED
HR85-8500-A
First tray ID indicator display field
Power canister standby power LED Power canister DC power LED Power canister service action LED
Controller B Power canister
Power canister AC power LED Power canister service required LED
Cray Private
63
Tiered Adaptive Storage™ Hardware Guide
DCBS (FC) Host Channels
Caution: To prevent electrostatic discharge damage to the tray, use proper ESD protection when you handle tray components.
•
Each controller in the controller-drive tray has four standard 8-GB/s Fibre Channel host ports.
•
Each controller in the controller-drive tray has the following type of host interface card (HIC): •
Four 8-Gb/s Fibre Channel host ports
Figure 4-26: DCBS (FC) Host Channels on the Controller-Drive Tray - Rear View Host interface card (HIC) Standard fibre channel host connectors
Standard fibre channel host connectors Host interface card (HIC)
64
Cray Private
SAS expansion connector InfiniBand host connectors
InfiniBand host connectors SAS expansion connector
HR85-8500-A
Hardware Implementation [4]
LEDs on the DCBS (FC) Controller-Drive Tray Figure 4-27: DCBS (FC) LEDs on the Left End Cap
Power LED Standby power LED Locate LED Overtemperature LED Service action required LED
Table 4-15: DCBS (FC) LEDs on the Left End Cap Location
HR85-8500-A
LED
Color
On
Off
1
Power
Green
Power is present.
Power is not present.
2
Standby power
Green
The controller-drive tray is in Standby Power mode.
The controller-drive tray is not in Standby Power mode.
3
Locate
White
Identifies a controllerdrive tray that you are trying to find.
Normal status.
4
Overtemperature
Amber
The temperature of the controller-drive tray has reached an unsafe level.
Normal status.
5
Service action required
Amber
A component within the controller-drive tray needs attention.
Normal status.
Cray Private
65
Tiered Adaptive Storage™ Hardware Guide
Figure 4-28: DCBS (FC) LEDs on the Controller Canister Main Faceplate Fibre channel link 1 LED 0 Fibre channel link 1 LED 1
Ethernet connector 2 link active LED
Fibre channel link 2 LED 0 Fibre channel link 2 LED 1
Ethernet connector 2 link rate LED
Fibre channel link 3 LED 0 Fibre channel link 3 LED 1
Ethernet connector 1 link active LED
Fibre channel link 4 LED 0 Fibre channel link 4 LED 1
Ethernet connector 1 link rate LED
Cache active LED
Battery service action required LED
Controller service action required LED Controller service action allowed LED
Battery charging LED
Table 4-16: DCBS (FC) LEDs on the Controller Canister Main Faceplate Location
Color
On
Off
1
Ethernet connector 1 link rate LED
Green
There is a 1000BASET rate.
There is a 10/100BASE-T rate.
2
Ethernet connector 1 link active LED
Green
The link is up (the LED blinks when activity is occurring).
The link is not active.
3
Ethernet connector 2 link rate LED
Green
There is a 1000BASET rate.
There is a 10/100BASE-T rate.
4
Ethernet connector 2 link active LED
Green
The link is up (the LED blinks when there is activity)
The link is not active.
5 and 6
Host connect 1 link active and data rate LED 0 off and LED 1 off
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 GB/s.
LED 0 off and LED 1 on
Green
Data rate 4 GB/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
7 and 8
66
LED
Host connection 2 link active and data rate
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-16: DCBS (FC) LEDs on the Controller Canister Main Faceplate Location
Color
On
Off
LED 0 off and LED 1 off
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 off and LED 1 on
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
LED 0 off and LED 1 of
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 on and LED 1 off
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
LED 0 off and LED 1 of
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 on and LED 1 off
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
13
Battery service action required LED
Amber
The battery in the controller canister has failed.
Normal status.
14
Battery charging LED
Green
The battery is fully charged. The LED blinks when the battery is charging.
The controller canister is operating without a battery or the existing battery has failed.
15
Controller service action allowed LED
Blue
The controller canister can be removed safely from the controllerdrive tray.
The controller canister cannot be removed safely from the controller-drive tray.
9 and 10
11 and 12
HR85-8500-A
LED
Host connection 3 link active and data rate
Host connection 4 link active and data rate
Cray Private
67
Tiered Adaptive Storage™ Hardware Guide
Table 4-16: DCBS (FC) LEDs on the Controller Canister Main Faceplate Location
LED
Color
On
Off
16
Controller service action required LED
Amber
A fault exists within the controller canister.
Normal status.
17
Cache active LED
Green
Cache is active.*
Cache is inactive the controller canister has been removed from the controller-drive tray.
* After an AC power failure, this LED blinks while cache offload is in process.
Figure 4-29: DCBS (FC) Right-Rear Subplate with an FC Host Interface Card SAS expansion active LED
FC Host Card
Lnk
2
8
4
2
Host interface card link 5 LED 1
68
8
7 FC host Host interface card interface card connector 7 link 6 LED 1 Host interface card link 6 LED 0
FC host interface card connector 6
Cray Private
4
Lnk
2
8
Lnk
4 Ch 8
FC host interface card connector 5
2
4 Ch 6
Ch 5 Host interface card link 5 LED 0
8
SAS expansion SFF-8088 port connector
SAS expansion fault LED
Mini USB
Drive Expansion 6Gb/s
S A S
FC host interface card connector 8 Host interface card link 8 LED 1 Host interface card link 8 LED 0 Host interface card link 7 LED 1 Host interface card link 7 LED 0
HR85-8500-A
Hardware Implementation [4]
Table 4-17: DCBS (FC) LEDs on the Controller Canister Fibre Channel Host Interface Card Subplates Location
Color
On
Off
2
Expansion fault LED
Amber
At least one of the four PHY is working, but another PHY cannot establish the same link to the device connected to the Expansion OUT connector.
Normal status.
3
Expansion active LED
Green
At least one of the four PHys in the OUt connector is working and a link has been made to the device connected to the Expansion connector.
The link is not active.
6 and 7
LED 0 off and LED 1 off
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 off and LED 1 on
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
LED 0 off and LED 1 off
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 off and LED 1 on
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
LED 0 off and LED 1 off
Green
The link is not active.
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 off and LED 1 on
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
LED 0 off and LED 1 off
Green
The link is not active.
9 and 10
12 and 13
15 and 16
HR85-8500-A
LED
Cray Private
69
Tiered Adaptive Storage™ Hardware Guide
Table 4-17: DCBS (FC) LEDs on the Controller Canister Fibre Channel Host Interface Card Subplates Location
LED
Color
On
LED 0 on and LED 1 off
Green
Data rate 2 Gb/s.
LED 0 off and LED 1 on
Green
Data rate 4 Gb/s.
LED 0 on and LED 1 on
Green
Data rate 8 Gb/s.
Off
Figure 4-30: DCBS (FC) LEDs on the Power Canister
Standby power LED DC power LED
AC power LED
Power service action allowed LED
Power service action required LED
Table 4-18: DCBS (FC) LEDs on the Power Canister Location
LED
Color
On
Off
1
Standby power
Green
The drive tray is in Standby mode and DC power is not available.
The drive tray is not in Standby mode and DC power is available.
2
DC power
Green
DC power from the power canister is available.
DC power from the power canister is not available.
3
Power service action allowed
Blue
The power canister can be removed safely from the drive tray.
The power canister cannot be removed safely from the drive tray.
4
Power service action required
Amber
A fault exists within the power canister.
Normal status.
5
AC power
Green
AC power to the power canister is present.
AC power to the power canister is not present.
Data Cache Block (DCBS) IB Interface, Controls, and Indicators •
70
The Data Cache Block Storage (DCBS) can be configured using a NetApp 5500-60 storage array with optional expansion trays or a standalone Netapp 2700-24 storage arrays with optional expansion trays (known as Metadata Block Storage (MDBS)). The controls and indicators show the NetApp 5500-60 only.
Cray Private
HR85-8500-A
Hardware Implementation [4]
•
The configuration of the host ports might appear different on your system depending on which host interface card configuration is installed. The controls and indicators might appear in different locations depending on which model is used for the CIMS, MDCs, etc.
•
The host ports are located on the host interface card (HIC). The following types of HICs are supported: •
Two InfiniBand connectors at 40 Gb/s
Figure 4-31: DCBS (IB) Host Channels on the Controllers with InfiniBand HIC - Rear View QSFP channels 1 and 2
QSFP channels 1 and 2
HR85-8500-A
Cray Private
71
Tiered Adaptive Storage™ Hardware Guide
LEDs on the DCBS (IB) Controller-Drive Tray Figure 4-32: DCBS (IB) LEDs on the Controller Canister Main Faceplate SAS channel 1 service action required LED
Ethernet port 2 link active LED
SAS channel 1 link up LED
Ethernet port 2 link rate LD
SAS channel 2 service action required LED
Ethernet port 1 link active LED
SAS channel 2 link up LED
Ethernet port 1 link rate LED
Tray ID indicator
Battery service action required LED
Cache active LED Controller service action required LED
10. Battery charging LED
Controller service action allowed LED
Table 4-19: DCBS (IB) LEDs on the Controller Canister Main Faceplate Location
72
LED
Color
On
Off
1
Ethernet port 1 link rate LED
Green
There is a 100BASE-T rate.
There is a 10BASE-T rate.
2
Ethernet port 1 link active LED
Green
The link is up (LED blinks when there is activity)
The link is not active.
3
Ethernet port 2 link rate LED
Green
there is a 100BASE-T rate.
There is a 10BASE-T rate.
4
Ethernet port 2 link active LED
Green
The link is up (the LED blinks when there is activity.)
The link is not active.
5
SAS channel 1 service action required LED
Amber
At least one of the four PHYs is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-19: DCBS (IB) LEDs on the Controller Canister Main Faceplate Location
LED
Color
On
Off
6
SAS channel 1 link up LED
Green
At least one of the four PHYs in the host interface port is working and a link exists to the device connected to the interface port connector.
A link error has occurred.
7
SAS channel 2 service action required LED
Amber
At least one four PHYs is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
8
SAS channel 2 link up LED
Green
At least one of the four PHYs in the host interface port is working and a link exists to the device connected to the interface port connector.
A link error has occurred.
9
Battery service action required LED
Amber
The battery in the controller canister has failed.
Normal status.
10
Battery charging LED
Green
The battery is fully charged. The LED blinks when the battery is charging.
The controller canister is operating without a battery or the existing battery has failed.
11
Controller service action allowed LED
Blue
The controller canister can be removed safely from the controllerdrive tray.
The controller canister cannot be removed safely from the controller-drive tray.
12
Controller service action required LED
Amber
A fault exists within the controller canister.
Normal status.
13
Cache active LED
Green
Cache is active.*
Cache is inactive or the controlled canister has been removed from the controllerdrive tray.
14
Tray ID indicator
N/A
N/A
N/A
* After an AC power failure, this LED blinks while cache offload is in process.
Figure 4-33: DCBS (IB) LEDs on the Controller Canister InfiniBand Host Interface Card
HR85-8500-A
Cray Private
73
Tiered Adaptive Storage™ Hardware Guide
Subplates QSFP channel 1 physical link up LED
QSFP channel 2 logical link up LED
QSFP channel 1 logical link up LED
QSFP channel 2 physical link up LED
Table 4-20: DCBS (IB) LEDs on the Controller Canister InfiniBand Host Interface Card Subplates* Location
74
LED
Color
On
Off
1
QSFP channel 1 logical link up LED
Amber
The link is up (LED blinks when there is activity.)
The controller has not yet loaded the driver for the ConnectX Host Channel Adapter.
2
QSFP channel 1 physical link up LED
Green
The physical link is active.
The physical link is not active.
3
QSFP channel 2 logical link up LED
Amber
The link is up (LED blinks when there is activity).
The controller has not yet loaded the driver for the ConnectX Host Channel Adapter.
4
QSFP Channel 2 physical link up LED
Green
The physical link is active.
The physical link is not active.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-34: DCBS (IB) LEDs on the Left Bezel
Power LED Standby power LED Controller-drive tray locate LED Controller drive tray overtemperature LED Service action required LED
Table 4-21: DCBS (IB) LEDs on the Left Bezel Location
HR85-8500-A
LED
Color
On
Off
1
Controllerdrive tray locate
White
Identifies a controller-drive tray that you are trying to find.
Normal status.
2
Service action required
Amber
A component within the controller-drive tray needs attention.
Normal status.
3
Controllerdrive tray overtemperature
Amber
The temperature of the controller-drive tray has reached an unsafe level.
Normal status.
4
Power
Green
Power is present.
Power is not present.
5
Standby power
Green
The controller-drive tray is in Standby Power mode.
The controller-drive tray is not in Standby Power mode.
Cray Private
75
Tiered Adaptive Storage™ Hardware Guide
Figure 4-35: DCBS (IB) LEDs on the Power Canister
Standby power LED DC power LED
AC power LED
Power service action allowed LED
Power service action required LED
Table 4-22: DCBS (IB) LEDs on the Power Canister Location
76
LED
Color
On
Off
1
Standby power
Green
The controller-drive tray is in Standby mode and DC power is not available.
The controller-drive tray is not in Standby mode and DC power is available.
2
DC power
Green
DC power from the powerfan canister is available.
DC power from the powerfan canister is not available.
3
Power service action allowed
Blue
The power-fan canister can be removed safely from the drive tray.
The power-fan canister cannot be removed safely from the drive tray.
4
Power service action required
Amber
A fault exists within the power-fan canister.
Normal status.
5
AC power
Green
AC power to the power-fan canister is present.
AC power to the power-fan canister is not present.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-36: DCBS (IB) LEDs on the Fan Canister
Fan power LED
Fan service action required ED
Fan service action allowed LED
Table 4-23: DCBS (IB) LEDs on the Fan Canister Location
HR85-8500-A
LED
Color
On
Off
1
Fan power
Green
Power to the fan canister is available.
Power to the fan customer-replaceable unit (CRU) is not available.
2
Fan service action required
Amber
A fault exists within the fan canister.
Normal status.
3
Fan service action allowed
Blue
The fan canister can be removed safely from the drive tray.
The fan canister cannot be removed safely from the drive tray.
Cray Private
77
Tiered Adaptive Storage™ Hardware Guide
ABS and DCBS LED Indicators Table 4-24: ABS and DCBS LED Symbols and General Behavior LED
Symbol
Location (Canisters)
Function
Power
Power-fan Interconnect-battery**
• On: The controller has power. • Off: The controller does not have power. Note: The controller canisters do not have a power LED. They receive their power from the power supplies inside the power-fan canisters.
Battery Fault
Battery
• On: The battery is missing or has failed. • Off: The battery is operating normally. • Blinking: The battery is charging.
Service Action Allowed
Drive (left LED, no symbol) Power-fan Controller Battery
On: You can remove the canister safely.
Service Action Required (Fault)
Drive
On: When the controller-drive tray LED is on, the cable is attached and at least one lane has a link up status, but at least one lane has a link down status. Off: One of the following conditions exists: • No cable is attached. • A cable is attached, and all lanes have a link up status. • A cable is attached, and all lanes have a link down status.
Service Action Required (Fault)
Controller Power-fan canister
On: The controller or the power-fan canister needs attention. Off: The controller and the power-fan canister are operating normally.
Service Action Required (Fault)*
FC Host Card
On: The FC cable is attached and at least one lane has a link up status, but at least one lane has a link down status. Off: One of the following conditions exists: • No cable is attached. • A cable is attached, and all lanes have a link up status. • A cable is attached, and all lanes have a link down status.
Locate
Front bezel on the controllerdrive tray* Front frame**
On: Assists in locating the tray.
78
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-24: ABS and DCBS LED Symbols and General Behavior LED
Symbol
Location (Canisters)
Function
InfiniBand Host Channel Connection* Host channel Connection (iSCSI)**
Controller
The status of the host channel is indicated: • “L” LED on: A link is established. • “A” LED on: Activity (data transfer) is present.
Fibre Channel Host Channel Connection*
Controller
The status of the host channel is indicated: • “2” LED on: A link is established. • “4” LED on: Activity (data transfer) is present.
Cache Active
Controller
The activity of the cache is indicated: • On: Data is in the cache. • Off: No data is in the cache.
Controller-Drive Tray Over-Temperature
Front bezel on the controllerdrive tray
On: The temperature of the controller-drive tray has reached an unsafe condition. Off: The temperature of the controller-drive tray is within operational range.
Controller-Drive Tray Standby Power
Front bezel on the controllerdrive tray
On: The controller-drive tray is in standby mode and the main AC/DC power is off. Off: The controller-drive tray is not in standby mode and the main AC/DC power is on.
Tray ID Indicator Diagnostic Display
Controller
The tray ID or a diagnostic code is indicated (see “Diagnostic Code Sequences for the Controller-Drive Trays” on page 120.) For example, if some of the cache memory dual inline memory modules (DIMMs) are missing in a controller, error code L8 appears in the diagnostic display (see “Lock-Down Codes for the Controller-Drive Trays” on page 119.)
AC Power
Power-fan Note: The LED is directly above or below the AC power switch and the AC power connector.
Indicates that the power supply is receiving AC power input.
HR85-8500-A
Cray Private
79
Tiered Adaptive Storage™ Hardware Guide
Table 4-24: ABS and DCBS LED Symbols and General Behavior LED
Symbol
Location (Canisters)
Function
DC Power (not available)
Power-fan Note: The LED is directly above or the below the DC power switch and the DC power connector.
Indicates that the power supply is receiving DC power input.
Ethernet Speed and Ethernet Activity
Controller
The speed of the Ethernet ports and whether a link has been established are indicated: • Left LED On: 1-GB/s speed. • Left LED Off: 100BASE-T or 10BASE-T speed. • Right LED On: A link is established. • Right LED Off: No link exists. • Right LED Blinking: Activity is occurring.
* Applicable to the NetApp 5460 only. ** Applicable to the NetApp 2724 and 5560 only.
Service Action Allowed LEDs Each controller canister, power-fan canister, and battery canister has a Service Action Allowed LED. The Service Action Allowed LED lets you know when you can remove a canister safely.
Caution: Never remove a controller canister, a power-fan canister, or a battery canister unless the appropriate Service Action Allowed LED is on.
Drive Channels ABS Drive Channels Each controller have levers for removing the controller from the controller-drive tray. The close spacing between the handle and the host connector might make it difficult to remove a cable that is attached to the host connector. If this problem occurs, use a flat-blade screwdriver to push in the release component on the cable connector.
80
•
The controller-drive tray supports the DE5600 drive tray for expansion.
•
Each controller has one dual-ported SAS expansion connector to connect to the drive trays.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-37: ABS Drive Channel Ports on the Controller-Drive Tray - Rear View Controller canister A
Controller canister B
SAS base host port (CH1 and CH2) SAS base host port (CH1 and CH2)
Figure 4-38: ABS Controller with Base Host Channel Ports (CH1 and CH2)
DCBS (FC) Drive Channels The E5400 controller-drive trays supports the DE6600 drive tray for expansion. •
Each controller has two SAS expansion connectors that must both be connected (one to the ESM A side and the other to the ESM B side) to the drive tray to ensure path redundancy.
Figure 4-39: Drive Channel Ports on the Controller-Drive Tray Controller canister A
Controller canister B
HR85-8500-A
Cray Private
Drive channel port (SAS expansion connector)
Drive channel port SAS drive expansion connector)
81
Tiered Adaptive Storage™ Hardware Guide
DCBS (IB) Drive Channels The E5500 controller-drive trays supports the DE6600 drive tray for expansion. •
Each controller has two SAS expansion connectors that must both be connected (one to the ESM A side and the other to the ESM B side) to the drive tray to ensure path redundancy.
Figure 4-40: DCBS (IB) Drive Channel Ports on the Controller-Drive Tray Controller canister A
SFF-8088 SAS channels (drive expansion)
Controller canister B
SFF-8008 SAS channels (drive expansion)
Figure 4-41: DCBS (IB) Controller with Labeled Drive Channel Ports (CH1 and CH2)
Disk Drive Trays ABS 24-bay Drive Tray Controls and Indicators Drive Trays The following disk drive types are used in the DE5600 drive enclosures:
82
•
600GB 10K SAS 2.5” HDDs
•
400GB or 800GB MLC 2.5” SSD drives
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-42: ABS Drive Tray - Front View Left end cap (has the drive tray LEDs)
Drives
Right end cap
Figure 4-43: ABS Drive Tray - Rear View
ESM A
SAS IN connector
SAS expansion (out) connector
SAS IN connector
ESM B
Figure 4-44: ABS Drive Tray with AC Power Option - Rear View Expansion Port SFF-8088 Connector 2 (IN)
Seven-Segment Display Indicators Serial Connector
Expansion Port SFF-8088 Connector 1 (IN)
Ethernet Connector Expansion Port SFF-8088 Connector (OUT)
ESM A Canister
Power-Fan Canister
HR85-8500-A
Power Connector
Cray Private
ESM B Canister
Power Switch
83
Tiered Adaptive Storage™ Hardware Guide
LEDs on the DE5400 Figure 4-45: ABS LEDs on the ESM Canister Host link 2 active LED
Host link 2 fault LED Host link 1 active LED
Ethernet link active LED Ethernet link rate LED
Host link 1 fault LED
ESM expansion link fault LED ESM expansion link active LED
ESM service action allowed LED Tray ID indicator
ESM service action required LED ESM power LED
Table 4-25: ABS LEDs on the ESM Canister Location
84
LED
Color
On
Off
1
Host link 1 fault
Amber
At least one PHY of the four connectors is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
2
Host link 1 active
Green
At least one of the four PHYs in the In port is working, and a link exists to the device connected to the host interface connector.
A link error has occurred.
3
Host link 2 fault
Amber
At least one PHY of the four connections is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
4
Host link 2 active
Green
At least one of the four PHYs in the In port is working, and a link exists to the device connected to the host interface connector.
A link error has occurred.
5
Ethernet link active
Green
The link is up. (The LED blinks when there is activity.)
The link is not active.
6
Ethernet link rate
Green
There is a 100BASE-T rate.
There is a 10BASE-T rate.
7
ESM expansion link fault
Amber
At least one of the four PHYs in the Out port is working, but another PHY cannot establish the same link to the Expansion Out connector.
Normal status.
8
ESM expansion link active
Green
At least one of the four PHYs in the Out port is working, and a link exists to the device connected to the Expansion Out connector.
A link error has occurred.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-25: ABS LEDs on the ESM Canister Location
LED
Color
On
Off
9
ESM service action allowed
Blue
The ESM can be removed safely from the drive tray.
The ESM cannot be removed safely from the drive tray.
10
ESM service action required
Amber
A fault exists within the ESM. (This LED defaults on at power up. This LED turns off after the software has completed its power up self-test sequence.)
Normal status.
11
ESM power
Green
12V power to the ESM is present.
Power is not present to the ESM.
12
Tray ID indicator
Green
See note.
Not applicable.
For more information about the tray ID indicators, see Chapter
5, "Block Storage Tray ID Codes," on
page 113. Figure 4-46: ABS LEDs on the AC Power-Fan Canister
Power-fan service action required LED Power-fan input AC power LED
Standby power LED Power-fan output DC power LED
Power-fan service action allowed LED
Table 4-26: ABS LEDs on the Power-Fan Canister Location
HR85-8500-A
LED
Color
On
Off
1
Standby power
Green
The drive tray is in Standby mode, and DC power is not available.
The drive tray is not in Standby mode, and DC power is available.
2
Power-fan DC power
Green
DC power from the powerfan canister is available.
DC power from the powerfan canister is not available.
3
Power-fan service action allowed
Blue
The power-fan canister can be removed safely from the drive tray.
The power-fan canister cannot be removed safely from the drive tray.
4
Power-fan service action required
Amber
A fault exists within the power-fan canister.
Normal status.
5
Power-fan AC power
Green
AC power to the power-fan canister is present.
AC power to the power-fan canister is not present.
Cray Private
85
Tiered Adaptive Storage™ Hardware Guide
DCBS 60-bay Drive Tray Controls and Indicators Figure 4-47: DCBS DE6600 Drive Tray - Rear View ESM A
SAS IN connectors
SAS expansion (out) connectors
ESM B
Figure 4-48: DCBS Maximum Capacity E-Series Array Configuration using DE6600 Shelves
86
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-49: DCBS LEDs on the Left Bezel
Power LED Standby power LED Controller-drive tray locate LED Controller drive tray overtemperature LED Service action required LED
Table 4-27: DCBS LEDs on the Left Bezel Location
HR85-8500-A
LED
Color
On
Off
1
Power
Green
Power is present.
Power is not present.
2
Standby power
Green
The drive tray is in Standby Power mode.
The drive tray is not in Standby Power mode.
3
Locate
White
Identifies a drive tray that you are trying to find.
Normal status.
4
Overtemperature
Amber
The temperature of the drive tray has reached an unsafe level.
Normal status.
5
Service action required
Amber
A component within the drive tray needs attention.
Normal status
Cray Private
87
Tiered Adaptive Storage™ Hardware Guide
Figure 4-50: DCBS LEDs on the ESM Canister Host link 2 active LED
Host link 2 fault LED Host link 1 active LED
Ethernet link active LED Ethernet link rate LED
Host link 1 fault LED
ESM expansion link fault LED ESM expansion link active LED
ESM service action allowed LED Tray ID indicator
ESM service action required LED ESM power LED
Table 4-28: DCBS LEDs on the ESM Canister Location
88
LED
Color
On
Off
1
Host link 1 fault
Amber
At least one PHY of the four connectors is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
2
Host link 1 active
Green
At least one of the four PHYs in the In port is working, and a link exists to the device connected to the host interface connector.
A link error has occurred.
3
Host link 2 fault
Amber
At least one PHY of the four connections is working, but another PHY cannot establish the same link to the device connected to the host interface port connector.
No link error has occurred.
4
Host link 2 active
Green
At least one of the four PHYs in the In port is working, and a link exists to the device connected to the host interface connector.
A link error has occurred.
5
Ethernet link active
Green
The link is up. (The LED blinks when there is activity.)
The link is not active.
6
Ethernet link rate
Green
There is a 100BASE-T rate.
There is a 10BASE-T rate.
7
ESM expansion link fault
Amber
At least one of the four PHYs in the Out port is working, but another PHY cannot establish the same link to the Expansion Out connector.
Normal status.
8
ESM expansion link active
Green
At least one of the four PHYs in the Out port is working, and a link exists to the device connected to the Expansion Out connector.
A link error has occurred.
9
ESM service action allowed
Blue
The ESM can be removed safely from the drive tray.
The ESM cannot be removed safely from the drive tray.
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-28: DCBS LEDs on the ESM Canister Location
LED
Color
On
Off
10
ESM service action required
Amber
A fault exists within the ESM. (This LED defaults on at power up. This LED turns off after the software has completed its power up self-test sequence.)
Normal status.
11
ESM power
Green
12V power to the ESM is present.
Power is not present to the ESM.
12
Tray ID indicator
Green
See note.
Not applicable.
For more information about the tray ID indicators, see “Tray ID Indicator Display for the ESMs on the Drive Trays” on page 121.
Figure 4-51: DCBS LEDs on the Power Canister
Standby power LED DC power LED
AC power LED
Power service action allowed LED
Power service action required LED
Table 4-29: DCBS LEDs on the Power-Fan Canister Location
HR85-8500-A
LED
Color
On
Off
1
Standby power
Green
The controller-drive tray is in Standby mode, and AC/DC power is not available.
The controller-drive tray is not in Standby mode, and AC/DC power is available.
2
Power-fan DC power
Green
DC power from the powerfan canister is available.
DC power from the powerfan canister is not available.
3
Power-fan service action allowed
Blue
The power-fan canister can be removed safely from the controller-drive tray.
The power-fan canister cannot be removed safely from the controller-drive tray.
4
Power-fan service action required
Amber
A fault exists within the power-fan canister.
Normal status.
5
Power-fan AC/DC power
Green
AC/DC power to the powerfan canister is present.
AC/DC power to the powerfan canister is not present.
Cray Private
89
Tiered Adaptive Storage™ Hardware Guide
Figure 4-52: DCBS LEDs on the Fan Canister
Fan power LED
Fan service action required ED
Fan service action allowed LED
Table 4-30: DCBS LEDs on the Fan Canister Location
90
LED
Color
On
Off
1
Fan power
Green
Power to the fan canister is available.
Power to the fan customerreplaceable unit (CRU) is available.
2
Fan service action required
Amber
A fault exists within the fan canister.
Normal status.
3
Fan service action allowed
Blue
The fan canister can be removed safely from the drive tray.
The fan canister cannot be removed safely from the drive tray.
Cray Private
HR85-8500-A
Hardware Implementation [4]
LEDs on the DE6600 Drive Drawers Figure 4-53: DCBS LEDs on the Drawer Drive drawer status service action required LED
Drive 4 activity LED Drive 5 activity LED
Drive drawer status service action allowed LED
Drive 6 activity LED
Drive 1 activity LED
Drive 7 activity LED
Drive 10 activity LED
Drive 2 activity LED
Drive 8 activity LED
Drive 11 activity LED
Drive 3 activity LED
Drive 9 activity LED
Drive 12 activity LED
Table 4-31: DCBS LEDs on the Drive Tray Drawer Location
LED
Color
On
Blinking
Off
1
Drive drawer service action required
Amber
An error has occurred.
Normal status.
2
Drive drawer service action allowed
Blue
The drive canister can be removed safely from the drive drawer in the drive tray.
The drive canister cannot be removed safely from the drive drawer in the drive tray.
3-14
Drive activity for drives 1 through 12 in the drive drawer
Green
The power is turned on, and the drive is operating normally.
Drive I/O activity is taking place.
The power is turned off.
Table 4-32: DCBS Drive State Represented by the LEDs Drive Activity LED (Green)
Drive State
HR85-8500-A
Drive Service Action Required LED (Amber)
Power is not applied.
Off
Off
Normal operation - The power is turned on, but drive I/O activity is not occurring.
On
Off
Normal operation - Drive I/O activity is occurring.
Blinking
Off
Service action required - A fault condition exists, and the drive is offline.
On
On
Cray Private
91
Tiered Adaptive Storage™ Hardware Guide
Figure 4-54: DCBS LEDs on the Drive
Drive service action allowed LED
Drive service action required LED
Table 4-33: LEDs on the Drives Location
LED
Color
On
Off
1
Drive service action allowed
Blue
The drive canister can be removed safely from the drive drawer in the drive tray.
The drive canister cannot be removed safely from the drive drawer in the drive tray.
2
Drive service action required
Amber
An error has occurred
Normal status.
Table 4-34: DCBS Drive State Represented by the LEDs Drive Activity LED (Green)
Drive State
92
Drive Service Action Required LED (Amber)
Power is not applied.
Off
Off
Normal operation - The power is turned on, but drive I/O activity is not occurring.
On
Off
Normal operation - Drive I/O activity is occurring.
Blinking
Off
Service action required - A fault condition exists, and the drive is offline.
On
On
Cray Private
HR85-8500-A
Hardware Implementation [4]
General Behavior of the LEDs on the DE6600 Drive Tray Table 4-35: DCBS LED Symbols and General Behavior LED
Symbol
Location (Canisters)
Function
Power
Drive tray (front bezel) ESM canister Power-fan canister
• On: Power is applied to the drive tray or the canister. • Off: Power is not applied to the drive tray or the canister.
Locate
Front bezel on the controller-drive tray
On or blinking: Indicates the drive tray that you are trying to find.
Controller-Drive Tray Over-Temperature
Front bezel on the controller-drive tray
On: The temperature of the drive tray has reached an unsafe condition. Off: The temperature of the drive tray is within operational range.
Controller-Drive Tray Standby Power
Front bezel on the controller-drive tray
On: The drive tray is in standby mode and the main DC power is off. Off: The drive tray is not in standby mode and the main DC power is on.
Service Action Allowed
ESM canister Power-fan canister Drive
On: It is safe to remove the ESM canister, the power-fan canister, or the drive. Off: Do not remove the ESM canister, the power-fan canister, or the drive. The drive has an LED but no symbol.
Service Action Required (Fault)
ESM canister Power-fan canister Drive
On: When the drive tray LED is on, a component within the drive tray needs attention. On: The ESM canister, the power-fan canister, or the drive needs attention. Off: The ESM canister, the power-fan canister, and the drive are operating normally. The drive has an LED but no symbol.
AC Power
ESM canister Power-fan canister
On: AC power is present. Off: AC power is not present.
Output DC power
Power-fan canister
On: Regulated DC output power from the power canister and the fan canister is present. Off: Regulated DC output power from the power-fan canister is not present.
HR85-8500-A
Cray Private
93
Tiered Adaptive Storage™ Hardware Guide
Table 4-35: DCBS LED Symbols and General Behavior LED
Symbol
Location (Canisters)
Function
Input DC power
Power-fan canister
On: Regulated DC input power from the power canister and the fan canister is present. Off: Regulated DC input power from the powerfan canister is not present.
Link Service Action Required (Fault)
ESM canister
On: The cable is attached and at least one lane has a linkup status, but one lane has a link-down status. Off: The cable is not attached, the cable is attached and all lanes have a link-up status, or the cable is attached and all lanes have a linkdown status.
ESM canister
On: The cable is attached and at least one lane has a linkup status. Off: The cable is not attached, or the cable is attached and all lanes have a link-down status.
Link up
94
Two LEDs above each expansion connector
Cray Private
HR85-8500-A
Hardware Implementation [4]
Tape Archive Storage Tier (TAST) TAS is normally configured with a TAST but it is not required. The Cray TAS sells and supports Spectralogic T50e, T200, T380, T680, and T950 tape libraries. However, TAS will support certain other tape libraries from Oracle(STK), Quantum, and IBM. Although Spectralogic tape drives are supported by Cray, they are serviced by Spectralogic. The TAS TAST is connected by FC-8 cables to the storage network. It supports T10k, TS1140, LTO-5, or LTO-6 media. The default offering for new Spectralogic tape library sales will be LTO-6. Cray will supply Spectralogic LTO-6 Terapacks. The TAST will support a minimum of 4 and a maximum of 16 tape drives based on the current MDC I/O. Each tape drives requires a FC8 connection to the SAN. Each group of 4 tape drives requires an FC8 port on each MDC.
Data Network Overview There are 2 types of switches, SAN and Network. The SAN switches may be fibre channel (QLogic) or infiniband (Mellanox). These switches provide a data storage fabric to connect the MDC's and DM's to the cache block storage, and to connect the MDC's to the robotic tape library. The network switches provide administrative communication between the CIMS and the MDC's and DM's. They provide access for system admins and all the servers (CIMS, MDC, and DM). In addition, the network switches provide metadata communication between the MDC’s and DM’s.
HR85-8500-A
Cray Private
95
Tiered Adaptive Storage™ Hardware Guide
InfiniBand Data Network Switches and Cables (Mellanox SX6036) Switch Platform Hardware Overview Figure 4-55: Mellanox SX6036 Switch System Power Side Panel
Figure 4-56: Mellanox SX6036 Switch System Connector Side Panel (Internally Managed)
Status LEDs The system status LEDs are located to the left of the connectors on the connector side panel, and on the power side at the far right. Table 4-36: Mellanox SX6036 System Status LEDs Symbol
96
Name
Description
Normal Conditions
Switch status LED
Shows the health of the switch
Solid green after 5 seconds
Fan status LED
Shows the health of the fans
Green
Power supply 1
Shows the health of the right side power supply unit
Green
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-36: Mellanox SX6036 System Status LEDs Symbol
Name
Description
Normal Conditions
Power supply 2
Shows the health of the left side power supply unit
Green, if installed for redundancy and connected to power source
Bad port LED
Lights up when a symbol error is detected
Off
Unit identifier LED
Lights up on command through the CLI
Off
Switch Status LED Figure 4-57: Mellanox Switch Status LED
Table 4-37: Switch Status LED Assignments LED Configuration
HR85-8500-A
Status/System Health LED
Solid green
The system is up and running
Flashing green
The system is booting up.This assignment is valid on managed systems only
Solid yellow
To be implemented in a future SW release
Solid red
Possible damage can result to the switch. Turn off immediately. e.g. bad firmware, can’t boot, overheated. If the system is booting up, it can take up to 5 minutes for the status LED to change to green.
Off
The system has no power.
Cray Private
97
Tiered Adaptive Storage™ Hardware Guide
Figure 4-58: Fan Status LED Connector Side
Table 4-38: Fan Status LED Assignments LED Configuration
Fan LED
Solid green
All fans are up and running
Solid red
Fan malfunction. The system should be powered down and troubleshoot the fan module.
Off
The fan unit is not receiving any power. Check that the fan unit is properly and completely inserted.
Power Supply Status LEDs Figure 4-59: Power Status LED Connector Side
This switch comes with two power supply units for redundancy. The SX6036 switch is available with one factory installed Power Supply Unit. A second Power Supply Unit can be added to support hot-swap ability and to add redundancy.
98
Cray Private
HR85-8500-A
Hardware Implementation [4]
Figure 4-60: Power Side Panel
The primary power supply unit (PS1) is located on the right side of the power side panel, with PS2 on the left side. Each PS unit has a single 2 color LED on the right side of the PS unit, that indicates the internal status of the unit. Figure 4-61: PS Unit Status LEDs
Table 4-39: PS Unit Status LED Assignments LED Color
HR85-8500-A
Status
Solid green
The power supply is delivering the correct voltage (12VDC)
Solid red
The PS unit is not operational
Off
There is no power to the system (neither PS unit is receiving power). If one PS unit is showing green and the second PS unit is unplugged, it will show a red indication
Cray Private
99
Tiered Adaptive Storage™ Hardware Guide
Bad Port LED Figure 4-62: Bad Port LED
Table 4-40: Bad Port LED Assignments LED Configuration
Description
Off
No symbol errors have bene received in the last few seconds
Flashing orange
One or possibly more ports have just received a symbol error. Possible causes are • Bad cable • Bad connection • Bad connector
UID LED Switch Identifier The UID LED is a debug feature that lists a blue LED on the switch connector side panel for ease in finding a particular switch within a cluster. Figure 4-63: Identifier LED
100
Cray Private
HR85-8500-A
Hardware Implementation [4]
Port Connector LEDs Above the downlink pots are two LEDs: one for the upper port and one for the lower port. Figure 4-64: Port LEDs
Table 4-41: Port Connector Physical and Logical Link Assignments for Ethernet Mode LED Configuration
LED Description
Off
Physical link is down/default
Solid green
Physical link is up with no traffic
Flashing green
Physical link is up with traffic
Flashing orange
Physical errors
Table 4-42: Connector Physical and Logical Link Assignments for IB Mode LED Status
LED Description
Off
No power to the port
Solid green
Logical link is up
Flashing green
Data activity flashing speed is proportional to data transfer speed
Solid orange
Physical link is up
Flashing orange
A problem with the physical link. Usually the SM is down or not enabling the port
In IB mode, the LED indicator, corresponding to each data port, will light orange when the physical connection is established (when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port). When a logical connection is made, the LED will change to green. When data is transferred, the light will blink green.
HR85-8500-A
Cray Private
101
Tiered Adaptive Storage™ Hardware Guide
Fibre Channel Data Network Switches and Cables (QLogic 5802V) Chassis Controls and LEDs Figure 4-65: QLogic 5802V Chassis Controls and LEDs Input Power LED (Green)
Heartbeat LED (Green) System Fault LED (Amber)
Table 4-43: QLogic 5802V Chassis Controls and LEDs LED
Color
On
Off
Input power
Green
Switch logic circuitry is receiving the correct DC voltages
Maintenance mode
Heartbeat
Green
Refer to “Heartbeat LED Blink Patterns” on page 103
Refer to “Heartbeat LED Blink Patterns” on page 103
System fault
Amber
A failure exists in the switch firmware or hardware (POST errors, over-temperature conditions or power supply malfunctions)
Normal status
•
Maintenance Button
Press the maintenance button to reset the switch, or to place the switch in maintenance mode. Maintenance mode sets the IP address to 10.0.0.1 and provides access to the switch for maintenance purposes when flash memory or the resident configuration file has been corrupted.
102
Cray Private
HR85-8500-A
Hardware Implementation [4]
Heartbeat LED Blink Patterns The Heartbeat LED indicates the operational status of the switch. When the POST completes with no errors, the Heartbeat LED blinks at steady rate of once per second. When the switch is in maintenance mode, the Heartbeat LED remains lit. All other blink patterns indicate critical errors. In addition to producing a heartbeat error blink patterns, a critical error also illuminates the System Fault LED. The Heartbeat LED shows an error blink pattern for the following conditions: •
1 blink—Normal operation
•
2 blinks—Internal Firmware Failure Blink Pattern
•
3 blinks—Fatal POST Error Blink Pattern
•
4 blinks—Configuration File System Error Blink Pattern
•
5 blinks—Over-Temperature Blink Pattern
FC Ports Figure 4-66: QLogic 5802V Fibre Channel Ports SFP Ports 0 1
10 11
2
12 13
3
4 5
14 15
XPAK Ports 6 7
16 17
8
18 19
9
20
21
22
23
Each SFP port is capable of 1, 2, 4, or 8Gbps transmission, depending on the SFP. SFP ports are hot-pluggable and can self-discover both the port type and transmission speed when connected to devices or other switches. Setting an SFP port to 1 Gbps that has an 8Gbps SFP transceiver will bring down the port. Figure 4-67: QLogic 5802V Port LEDs Logged-In LED Activity LED
HR85-8500-A
Logged-In LED Activity LED
Cray Private
103
Tiered Adaptive Storage™ Hardware Guide
Table 4-44: QLogic 5802V Port LEDs LED
Color
On
Off
Port logged-in
Green
Port initialized or logged in Refer to Logged-In LED Indications
Port connection is broken or an error occurs that disables the port Refer to Logged-In LED Indications
Port activity
Green
Data is passing through the port
Data is not passing through the port
Logged-In LED Indications The Logged-In LED has three indications: •
Continuous illumination: A device is logged in to the port.
•
Flashing once per second: A device is logging in to the port, or the port is in the diagnostics state.
•
Flashing twice per second: The port is down, offline, or an error has occurred.
If a Logged-In LED is flashing twice per second, review the event browser for alarm messages regarding the affected port. You can also inspect the alarm log using the Show Alarm command. If there is an error, alarm messages may point to one or more of the following conditions: •
E_Port Isolation
•
Excessive Port Errors
Ethernet Port The Ethernet port is an RJ-45 connector that provides a connection to the management workstation through a 10/100 Base-T Ethernet cable. Figure 4-68: QLogic 5802V Ethernet Port Activity LED
104
Link Status LED
Cray Private
HR85-8500-A
Hardware Implementation [4]
Table 4-45: QLogic 5802V Power Supply ED LED
Color
On
Off
Link status
Green
Ethernet connection is established
No ethernet connection is established
Activity
Green
Data is transmitted and received over Ethernet connection
No data is transmitted and received over Ethernet connection
Serial Port Figure 4-69: QLogic 5802V Serial Port
1 6
5 9
RS-232 Connector Pin Identification
Serial Port
Power Supplies and Fans The model 5802V switch has two, hot-pluggable power supplies that convert 100–240 VAC to DC voltages for the various switch circuits. Each power supply has an AC power receptacle and two status LEDs. Figure 4-70: Model 5802V Power Supplies Status LED (Green)
AC Power Receptacle
Fault LED (Amber)
Power Supply 1
HR85-8500-A
Power Supply 2
Cray Private
105
Tiered Adaptive Storage™ Hardware Guide
Table 4-46: QLogic 5802V Power Supply LEDs LED
106
Color
On
Off
Power supply fault
Amber
A power supply fault exists and requires attention
Normal status
Power supply status
Green
Power supply is receiving AC voltage and producing the correct DC voltages
Power supply is not receiving AC voltage or not producing the correct DC voltages
Cray Private
HR85-8500-A
Hardware Implementation [4]
Ethernet Management Network Switches and Cables (Dell PowerConnect 5548) Dell PowerConnect 5548 Ports Figure 4-71: PowerConnect 5548 Ports
HR85-8500-A
Cray Private
107
Tiered Adaptive Storage™ Hardware Guide
Buttons and LEDs Figure 4-72: Dell PowerConnect 5548 Front LED Panel
Table 4-47: Dell PowerConnect 5548 Front Panel LEDs LED
Color
Power supply (PWR)
Green
Static: The switch is on Flashing: The Locator function is enabled
Status
Green or Red
Green static: The switch is operating normally Green flashing: The switch is booting Red static: A critical system error has occurred Red flashing: A noncritical system error has occurred
Stacking number Modular/redunda ncy power supply (MPS/RPS)
108
On
Off The switch is turned off
Indicates the unit ID of the device in stack Green or Red
Cray Private
Green static: The MPS/RPS is currently operating Red static: The MPS/RPS failed
The MPS/RPS is not plugged in
HR85-8500-A
Hardware Implementation [4]
Table 4-47: Dell PowerConnect 5548 Front Panel LEDs LED
HR85-8500-A
Color
On
Locator
Green
Flashing: Locator function is enabled Static: Locator function is disabled
Master
Green
Static: The device is a master unit
Fan (FAN)
Green or Red
Green static: All device fans are operating normally Red static: One or more of the device fans are not operating
Cray Private
Off
The device is not a master unit
109
Tiered Adaptive Storage™ Hardware Guide
Gigabit Ports Figure 4-73: Giga Port LEDs
LNK
ACT/PoE
LNK
ACT/PoE
Table 4-48: Giga Ports on non-POE Enabled Devices LEDs LED
Color
On
Off
LNK
Green, Yellow, or Amber
Green flashing: Link is up and the port is either transmitting or receiving at 1000 Mbps Solid green: Link is up at high speeds Yellow flashing: Link is up and the pot is either transmitting or receiving data at 100 Mbps Solid amber: Link is up at lower speeds
The port is currently not operating
ACT
Green
Green flashing: There is activity on the port
There is no activity on the port
HDMI Port LEDs The HDMI ports have a Speed/link (LNK) LED on their left side and an activity (ACT) LED on their right side. Table 4-49: HDMI (Stacking) Port LEDs LED
110
Color
On
Off
Speed/link
Green
Solid green: Port is linked to device
Port is currently not operating
ACT
Green
Flashing green: Port is either transmitting or receiving
Port is not transmitting or receiving
Cray Private
HR85-8500-A
Hardware Implementation [4]
SFP LEDs The SFP+ ports each have two LEDs, marked as LNK and ACT, associated with them. Table 4-50: SFP Port LEDs LED
HR85-8500-A
Color
On
LNK
Green or Amber
Solid green: Link is at highest speed Solid amber: Link is at lowest speed
ACT
Green
Flashing green: Port is either transmitting or receiving
Cray Private
Off Port is currently not linked
111
Tiered Adaptive Storage™ Hardware Guide
112
Cray Private
HR85-8500-A
Block Storage Tray ID Codes [5] ABS (NetApp 2700) and DCBS (NetApp 5500) Tray ID Indicator Codes Tray ID Failure Codes The tray ID indicator displays values that can identify a failing component. Table 5-51: Storage Controller Tray ID Failure Codes Component
Tray ID Indicator Display Used in this Controller State
(Code) (See footnote 1)
Power-on Diagnostics
Suspended
Operational
Processor DIMM (dx +) (See footnote 2)
yes
yes
no
Cache DIMM (CX +)
yes
yes
no
Host card (HX +)
yes
yes
yes
Flash drive (FX +)
no
no
yes
Category delimiter (--) After (--), a “CF” (for Component Failure) is displayed, followed by one of the codes specified above.
The separator between categorydetail code pairs is used when more than one category detail code pair exists in the sequence.
End-of-sequence delimiter Appears as a blank space. (See footnote 3)
The end-of-sequence delimiter is automatically inserted by the hardware at the end of a code sequence.
1. A two-digit code that starts a dynamic display sequence. 2. A two-digit code that follows the category code with more specific information 3. The plus (+) sign indicates that a two-digit code display with the Diagnostic LED on.
HR85-8500-A
Cray Private
113
Tiered Adaptive Storage™ Hardware Guide
Controller-Drive Tray Sequence Code Definitions During normal operation, the tray ID display on each controller canister displays the controller-drive tray ID. The Diagnostic LED (lower-digit decimal point) comes on when the display is used for diagnostic codes and goes off when the display is used to show the tray ID. Table 5-52: Controller Drive Tray Sequence Code Definitions Category
Category Code (See Footnote 1)
Detail Codes (See Footnote 2)
Startup error
SE+ (See footnote 3)
• 88+ Power-on default • dF+ Power-on diagnostic fault • Sx Power-on validation error
Operational error
OE+
Lx+ Lock-down codes. (See “Lock-Down Codes for the Controller-Drive Tray” on page 114 and “Tray ID Indicator Display Use Cases” on page 116.)
Operational state
OS+
• • • •
OL+ = Offline. bb+ = Battery backup (operating on batteries). OH + CPU temperature exceeds the warning level. CF+ = Component failure.
Component failure
CF+
• • • • • •
dx+ = Processor or cache DIMM. Cx = Cache DIMM. Px+ = Processor DIMM. Hx+ = Host interface card. Fx+ = Flash drive. bl + Base controller card.
Diagnostic failure
dE+
Lx+ = Lock-down codes (See “Lock-Down Codes for the Controller-Drive Tray” on page 114 and “Tray ID Indicator Display Use Cases” on page 116.)
Category delimiter
dash+
The separator between category-detail code pairs is used when more than one category detail code pair exists in the sequence.
End-of-sequence delimiter
blank - (See footnote 4)
The end-of-sequence delimiter is automatically inserted by the hardware at the end of a code sequence.
1.A two-digit code that starts a dynamic display sequence. 2. A two-digit code that follows the category code with more specific information. 3. The plus (+) sign indicates that a two-digit code displays with the Diagnostic LED on. 4. No codes display, and the Diagnostic LED is off.
Lock-Down Codes for the Controller-Drive Tray Use the following table to determine the diagnostic lock-down code definitions on the tray ID indicator display in the controller canister. In general, these diagnostics appear only when the controller is in a non-operational state. The controller can be in a non-operation state as a result
114
Cray Private
HR85-8500-A
Block Storage Tray ID Codes [5]
of a configuration problem (such as mismatched controller types) or as a result of a hardware fault. If the amber Controller Service Action Required LED is on, the non-operational state is the result of a hardware fault. Table 5-53: Supported Diagnostic Lock-Down Codes on the Tray ID Indicator Display Diagnostic Code
Description
-- --
The firmware is booting.
L0
The controller types are mismatched, which result in a suspended controller state.
L2
A persistent memory error has occurred, which results in a suspended controller state.
L3
A persistent hardware error has occurred, which results in a suspended controller state.
L4
A persistent data protection error has occurred, which results in a suspended controller state.
L5
An autocode synchronization (ACS) failure has been detected, which results in a suspended controller state.
L6
An unsupported host interface card has been detected, which results in a suspended controller state.
L7
A sub-model identifier either has not been set or has been mismatched, which results in a suspended controller state.
L8
A memory configuration error has been occurred, which results in a suspended controller state.
L9
A link speed mismatch condition detected in either the ESM or power supply, causing suspended controller state.
Lb
A host interface card configuration error has been detected, which results in a suspended controller state.
LC
A persistent cache backup configuration error has been detected, which results in a suspended controller state.
Ld
A mixed cache memory DIMMs condition has been detected, which results in a suspended controller state.
LE
Uncertified cache memory DIMM sizes have been detected, which result in a suspended controller state.
LF
The controller has locked down in a suspended state with limited symbol support.
LH
A controller firmware mismatch has been detected, which results in a suspended controller state.
LJ
The controller does not have enough memory to support the configuration, causing a suspended controller state.
LL
The controller cannot access either midplane SBB EEPROM, which results in a suspended controller state.
Ln
A canister is not valid for a controller, which results in a suspended controller state.
LP
Drive port mapping tables are not detected, which results in a suspended controller state.
Lr
A non field replaceable unit has been replaced, which results in a suspended controller state.
Lt
A configuration data base corruption, which results in a suspended controller state.
LU
The start-of-day (SOD) reboot limit has been exceeded which results in a suspended controller state.
S1*
The controller detects a checksum failure in EEPROM, which results in a suspended controller state.e
S2*
The SBB Signature/Revision is invalid, which results in a suspended controller state.
S3*
An unsupported enclosure is detected in a storage array, which results in a suspended controller state.
S4*
The power supply is incapable of powering the controller, which results in a suspended controller state.
S5*
The SBB pairing fails, which results in a suspended controller state.
* This lockdown code is for the NetApp 2700 only.
HR85-8500-A
Cray Private
115
Tiered Adaptive Storage™ Hardware Guide
Tray ID Indicator Display Use Cases Use the following table displays both startup errors and operation errors that can be used to point to a component failure. Table 5-54: Tray ID Indicator Display Uses Cases with Repeating Sequences Uses Cases
Repeating Sequence
Use Case: Controller power-on • Normal power-on controller insertion. • Controller inserted while held in reset.
SE+ 88+ blank-
Uses Cases
Repeating Sequence
Use Case: Operational states • Normal operation.
xy- (static controller tray ID)
• Start-of-day (SOD) processing
OS+ Sd+ blank-
• The controller is placed in reset while displaying the tray ID.
OS+ OL+ blank-
• The controller is operating on batteries (cache backup).
OS+ bb+ blank-
• The CPU temperature has exceeded the warning level.
OS+ OH+ blank-
Uses Cases
Repeating Sequence
Use Case: Component failure when the controller is operational. • Failed host interface card
OS+ CF+ HX+ blank-
• Failed flash drive
OS+ CF+ Fx+ blank-
Uses Cases
Repeating Sequence
Use case: Power-on diagnostic failure • A component that is not a field replaceable unit failure has been detected.
SE+ dF+ blank-
• A processor DIMM failure has been detected.
SE+ dF+ “- -”+ CF+ Px+ blank-
• A cache memory DIMM failure has been detected.
SE+ dF+ “- -”+ CF+ Cx+ blank-
• A processor DIMM or a cache memory DIMM failure has been detected.
SE+ dF+ “- -”+ CF+ dx+ blank-
• A host interface card failure has been detected.
SE+ dF+ “- -”+ CF+ Hx+ blank-
• An incorrect number of cache backup devices have been detected.
SE+ LC+ “- -”+ CF+ FX+ blank-
Uses Cases
Repeating Sequence
Use Case: Controller is suspended and there are no other errors to report. • All lockdown conditions
116
OE+ Lx+ blank-
Cray Private
HR85-8500-A
Block Storage Tray ID Codes [5]
Table 5-54: Tray ID Indicator Display Uses Cases with Repeating Sequences (continued) Uses Cases
Repeating Sequence
Use Case: The controller is suspended because of component errors. • Persistent processor DIMM error correcting code (ECC) errors have been detected.
OE+ L2+ “- -”+ CF+ CX+ blank-
• Persistent cache DIMM ECC errors have been detected.
OE+ L2+ “- -”+ CF+ CX+ blank-
• Persistent processor or cache DIMM ECC errors have been detected.
OE+ L2+ “- -”+ CF+ CX+ blank-
Uses Cases
Repeating Sequence
Use Case: The controller has been suspended as a result of persistent cache backup configuration errors. • The write-protect switch is set during cache restore.
OE+ LC+ blank-
• The memory size changed with dirty data in the flash drives.
OE+ LC+ dd+ blank-
Uses Cases
Repeating Sequence
Use case: The controller has been suspended as a result of diagnostic errors. • A cache memory diagnostic failure has been detected.
dE+ L2+ “- -”+ CF+ Cx+ blank-
• A base controller diagnostic failure has been detected
dE+ L3+ “- -”+ CF+ b1+ blank-
• A base controller IOC diagnostic failure has been detected.
dE+ L3+ “- -”+ CF+ b2+ blank-
Tray ID Indicator Display for the Drive Trays During normal operation, the tray ID display on each ESM displays the drive tray ID. The diagnostic LED (lower-digital decimal point) comes on when the display is used for diagnostic codes and goes off when the display is used to show the tray ID. Table 5-55: Supported Diagnostic Codes on the ESM Diagnostic Code
HR85-8500-A
Description
-- --
The firmware is booting.
.8, 8., 88
The ESM is being held in reset by another ESM.
AA
ESM A firmware is in the process of booting (the diagnostic indicator is not yet set).
bb
ESM B firmware is in the process of booting (diagnostic indicator is not yet set).
L0
The ESM types are mismatched.
L2
A persistent memory error has occurred.
L3
A persistent hardware error has occurred.
L9
An over-temperature condition has been detected in either the ESM or the power supply.
LL
The midplane SSB VPD EEPROM cannot be accessed.
Ln
The ESM canister is not valid for this drive tray.
Cray Private
117
Tiered Adaptive Storage™ Hardware Guide
Table 5-55: Supported Diagnostic Codes on the ESM (continued) Diagnostic Code
Description
LP
Drive port mapping tables are not found.
H2
The ESM configuration is invalid or incomplete, and it operates in Degraded state.
H3
The maximum number of ESM reboot attempts has been exceeded.
H4
This ESM cannot communicate with the alternate ESM.
H5
A midplane harness failure has been detected in the drive tray.
H6
An ESM firmware failure has been detected.
H9
A non-catastrophic hardware failure has occurred. The ESM is operating in a Degraded state.
J0
The ESM canister is incompatible with the drive tray firmware.
DCBS (NetApp 5400) Tray ID Indicator Codes Sequence Code Definitions for the Controller-Drive Trays During normal operation, the tray ID display on each controller canister displays the controllerdrive tray ID. The Diagnostic LED (lower-digit decimal point) comes on when the display is used for diagnostic codes and goes off when the display is used to show the tray ID. Table 5-56: Sequence Code Definitions for the 5400 Controller-Drive Trays Category
Category Code (See Footnote 1)
Detail Codes (See Footnote 2)
Startup error
SE+ (see footnote 3)
• 88+ Power-on default • dF+ Power-on diagnostic fault • Sx Power-on Validation error
Operational error
OE+
Lx+ Lock-down codes (See footnote 3.)
Operational state
OS+
• • • •
OL+ = Offline. (See footnote 11.) bb+ = Battery backup (operating on batteries). OH+ = CPU temperature exceeds the warning level. CF+ = Component failure. (See footnote 12.)
Component failure
CF+
• • • • • •
dx+ = Processor or cache DIMM. (See footnote 6.) Cx = Cache DIMM (See footnote 7.) Px+ = Processor DIMM. (See footnote 8.) Hx+ = Host interface card. Fx+ = Flash drive. bl + Base Controller Card
Diagnostic failure
dE+
Lx+ = Lock-down code. (See footnote 3.)
118
Cray Private
HR85-8500-A
Block Storage Tray ID Codes [5]
Table 5-56: Sequence Code Definitions for the 5400 Controller-Drive Trays Category Code (See Footnote 1)
Category
Detail Codes (See Footnote 2)
Category delimiter
dash+
the separator between categorydetail code pairs is used when more than one category detail code pair exists in the sequence. (See footnote 4 and footnote 9.)
End-of-sequence delimiter
blank -
The end-of-sequence delimiter is automatically inserted by the hardware at the end of a code sequence. (see footnote 5 and footnote 10.)
1. xy+ 2-digit code with the Diagnostic light ON. 2. xy- 2-digit code with the Diagnostic light OFF. 3. Lx+ Lock-down codes (see ____________ for a list of the lock-down codes.) 4. dash+ All segments off except for the middle segments and with the Diagnostic light ON. 5. blank- All segments off with the Diagnostic light OFF. 6. dx+ USed when there is a single memory system for processor and data cache. 7. Cx+ USed when there are separate processor and data cache memory systems. 8. Px+ USed when there are separate processor and data cache memory systems. 9. Category- Detail separator used when there is more than one category-detail code pair in the sequence. 10. End-of-sequence indicator automatically inserted by hardware at the end of the sequence. Example: SE+ 88+ blank- (repeat) 11. If a tray ID is being displayed, this sequence is programmed to display if the controller is subsequently held in reset. 12. The tray ID is normally displayed during normal operation. This operational state is displayed if an internal controller component failure occurs while the controller is online. An additional detail code identifies the failed component as defined for the Component Failure category. This sequence continues to be displayed even if the controller is subsequently placed offline (held in reset) to service the failed component.
Lock-Down Codes for the Controller-Drive Trays Use the following table to determine the diagnostic lock-down code definitions on the tray ID indicator display in the controller canister. In general, these diagnostics display only when the controller is in a non-operational state. The controller can be in a non-operation state as a result of a configuration problem (such as mismatched controller types) or as a result of a hardware fault. If the amber Controller Service Action Required LED is on, the non-operational state is the result of a hardware fault. Table 5-57: Supported Diagnostic Lock-Down Codes on the Tray ID Indicator Display Diagnostic Code
Description
L0
The controller types are mismatched, which result in a suspended controller state.
L1
Missing interconnect battery canister. (This code is not applicable for the E5400.)
L2
A persistent memory error has occurred, which results in a suspended controller state.
L3
A persistent hardware error has occurred, which results in a suspended controller state.
L4
A persistent data protection error has occurred, which results in a suspended controller state.
L5
An auto-code synchronization (ACS) failure has been detected, which results in a suspended controller state.
L6
An unsupported host interface card has been detected, which results in a suspended controller state.
L7
A sub-model identifier either has not been set or has been mismatched, which results in a suspended controller state.
L8
A memory configuration error has occurred, which results in a suspended controller state.
L9
A link speed mismatch condition has been detected in either the ESM or the power supply, which results in a suspended controller state.
HR85-8500-A
Cray Private
119
Tiered Adaptive Storage™ Hardware Guide
Table 5-57: Supported Diagnostic Lock-Down Codes on the Tray ID Indicator Display Diagnostic Code
Description
LA
Reserved
Lb
A host interface card configuration error has been detected, which results in a suspended controller state.
LC
A persistent cache backup configuration error has been detected, which results in a suspended controller state.
Ld
A mixed cache memory DIMMs condition has been detected, which results in a suspended controller state.
LE
Uncertified cache memory DIMM sizes have been detected, which result in a suspended controller state.
LF
The controller has locked down in a suspended state with limited symbol support.
LH
A controller firmware mismatch been detected, which results in a suspended controller state.
LL
The controller cannot access either midplane SBB EEP-ROM, which results in a suspended controller state.
Ln
A canister is not valid for a controller, which results in a suspended controller state.
LP
Drive port mapping tables are not detected, which results in a suspended controller state.
Lr
The controller has detected a component that is not field replaceable unit (FRU). Please contact your Technical Support representative.
LU
The start-of-day (SOD) reboot limit has been exceeded, which results in a suspended controller state.
Diagnostic Code Sequences for the Controller-Drive Trays Use the following table to determine the code sequences on the tray ID indicator display in the controller canister for the E5400 controller-drive trays. These repeating sequences can be used to diagnose potential problems with the controller tray. Table 5-58: Diagnostic Code Sequences for the E5400 Controller-Drive Trays Displayed Diagnostic Code Sequences
Description
SE+ 88+ blank-
One of the following power-on conditions exists: • Controller power-on • Controller insertion • Controller inserted while held in reset
xy -
Normal operation.
OS+ Sd+ blank-
Start-of-day (SOD) processing.
OS+ OL+ blank-
The controller is placed in reset while displaying the tray ID.
OS+ bb+ blank-
The controller is operating on batteries (cache backup).
OS+ OH+ blank-
The CPU temperature exceeds the warning level.
OS+ CF+ Hx+ blank-
A failed host interface card has been detected (See footnote 1 and footnote 2.)
OS+ CF+ Fx+ blank-
A failed flash drive has been detected (see footnote 1 and footnote 2.)
SE+ dF+ blank-
A non-replaceable component failure has been detected (See footnote 1.)
SE+ dF+ dash+ CF+ Px+ blank-
A processor DIMM failure has been detected. (See footnote 1.)
SE+ dF+ dash+ CF+ Cx+ blank-
A cache memory DIMM failure has been detected. (See footnote 1.)
SE+ dF+ dash+ CF+ dx+ blank-
A processor or cache DIMM failure has been detected. (See footnote 1.)
120
Cray Private
HR85-8500-A
Block Storage Tray ID Codes [5]
Table 5-58: Diagnostic Code Sequences for the E5400 Controller-Drive Trays Displayed Diagnostic Code Sequences
Description
SE+ dF+ dash+ CF+ Hx+ blank-
A host card failure has been detected. (See footnote 1.)
SE+ LC+ dash+ CF+ Fx+ blank-
A lockdown condition has been detected.
OE+ L2+ dash+ CF+ Px+ blank-
Persistent processor DIMM ECC errors have been detected, which result in a suspended controller state.
OE+ L2+ dash+ CF+ Cx+ blank-
Persistent cache DIMM ECC errors have been detected, which result in a suspended controller state.
OE+ L2+ dash+ CF+ dx+ blank-
Persistent processor or cache DIMM ECC errors have been detected, which result in a suspended controller state.
OE+ LC+ blank-
The write-protect switch is set during cache restore, which results in a suspended controller state.
OE+ LC+ dd+ blank-
The memory size is changed from bad data in the flash drives, which results in a suspended controller state.
DE+ L2+ dash+ CF+ Cx+ blank-
A cache memory diagnostic has failed, which results in a suspended controller state.
DE+ L3+ dash+ CF+ bl+ blank-
A base controller diagnostic has failed, which results in a suspended controller state.
DE+ L3+ dash+ CF+ b2+ blank-
An I/O controller (IOC) diagnostic has failed for the chip on the controller board, which results in a suspended controller state.
1. If the controller detects more than one component failure, only the first component failure detected appears in the tray ID indicator display. 2. If this component failure displays in the tray ID indicator while the controller is operational, the storage management software will also generate event notifications, such as a MEL event, for the same failure condition.
Tray ID Indicator Display for the ESMs on the Drive Trays During normal operation, the tray ID display on each ESM displays the drive tray ID. The Diagnostic LED (lower-digit decimal point) comes on when the display is used for diagnostic codes and goes off when the display is used to show the tray ID. Note: If a power-on or reset occurs, the Diagnostic LED, the Heartbeat LED (upper-digit decimal point), and all tray ID indicators of both digits come on. The Diagnostic LED remains on until the drive tray ID appears. Table 5-59: Supported Diagnostic Codes Diagnostic Code
ESM State
Description
-- --
Suspended
The firmware is booting.
.8, 8., or 88
Suspended
This ESM is being held in reset by another ESM
AA
Suspended
ESM A firmware is in the process of booting (the diagnostic indicator is not yet set).
bb
Suspended
ESM B firmware is in the process of booting (the diagnostic indicator is not yet set).
L0
Suspended
The ESM types are mismatched.
L2
Suspended
A persistent memory error has occurred.
L3
Suspended
A persistent hardware error has occurred.
HR85-8500-A
Cray Private
121
Tiered Adaptive Storage™ Hardware Guide
Table 5-59: Supported Diagnostic Codes Diagnostic Code
ESM State
Description
L9
Suspended
An over-temperature condition has been detected in either the ESM or the power supply.
LL
Suspended
The midplane SBB VPD EEPROM cannot be accessed.
Ln
Suspended
The ESM canister is not valid for this drive tray.
LP
Suspended
Drive port mapping tables are not found.
H2
Suspended
The ESM configuration is invalid or incomplete, and it operates in a Degraded state.
H3
Suspended
The maximum number of ESM reboot attempts have been exceeded.
H4
Suspended
This ESM cannot communicate with the alternate ESM.
H5
Suspended
A midplane harness failure has been detected in the drive tray.
H6
Suspended
An ESM firmware failure has been detected.
H7
ESM is operating as a rate other than the switch setting.
H9
A non-catastrophic hardware failure has occurred. The ESM is operating in a Degraded state.
J0
122
Suspended
The ESM canister is incompatible with the drive tray firmware.
Cray Private
HR85-8500-A
