Ibm System Z Connectivity Handbook

Transcript

Front cover Draft Document for Review July 21, 2010 6:06 pm SG24-5444-11 IBM System z Connectivity Handbook The connectivity options available for your data center, and beyond Comparisons of the supported protocols and architectures Guidance for typical usage of the connectivity options Bill White Mario Almeida Erik Bakker Vicente Ranieri Jr Jin Yang ibm.com/redbooks Draft Document for Review July 21, 2010 6:06 pm 5444edno.fm International Technical Support Organization IBM System z Connectivity Handbook August 2010 SG24-5444-11 5444edno.fm Draft Document for Review July 21, 2010 6:06 pm Note: Before using this information and the product it supports, read the information in “Notices” on page xi. Twelfth Edition (August 2010) This edition applies to the connectivity options available on the IBM zEnterprise 196, IBM System z10, and System z9 servers. © Copyright International Business Machines Corporation 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Draft Document for Review July 21, 2010 6:06 pm 5444edno.fm iii 5444edno.fm iv IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444TOC.fm Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii xiii xiv xiv Chapter 1. Connectivity introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Channel Subsystem (CSS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Parallel channel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Enterprise Systems Connection (ESCON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Fibre Connection (FICON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.6 Open Systems Adapter-Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.7 HiperSockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.8 Parallel Sysplex and coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.9 System z channel feature support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Chapter 2. Channel Subsystem overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 CSS description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 CSS elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Multiple CSS concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Multiple CSS structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Multiple Image Facility (MIF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.5 Physical Channel ID (PCHID) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.6 Adapter ID (AID) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.7 Multiple CSS construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.8 Channel spanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.9 Multiple subchannel sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.10 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 I/O configuration management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 IBM Configurator (eConfig) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Hardware Configuration Definition (HCD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 CHPID Mapping Tool (CMT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 I/O configuration planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 14 14 14 15 16 17 19 20 21 23 24 24 25 25 25 26 29 Chapter 3. Parallel channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 ESCON conversion to parallel channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Parallel channel - single path to a control unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Parallel channel modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Parallel channel elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Parallel channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Parallel channel control unit types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 32 32 33 33 34 34 35 35 Chapter 4. Enterprise Systems Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. v 5444TOC.fm Draft Document for Review July 21, 2010 6:06 pm 4.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 ESCON modes and topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 ESCON elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Multiple Image Facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 ESCON channel support on System z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 38 39 43 43 44 46 47 Chapter 5. Fibre Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 FICON modes and topologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 FCP Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 FCP and FICON mode characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 FICON elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 FICON Express8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 FICON Express4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 FICON Express2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 FICON Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.5 Qualified FICON and FCP products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.6 FICON advantages over ESCON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.7 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.8 Resource Measurement Facility (RMF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 50 50 53 58 61 73 75 75 77 78 79 79 80 80 81 Chapter 6. Channel-to-channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 ESCON CTC channel connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 FICON (FCV)-to-ESCON CTC connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.3 FICON channel-to-channel (FCTC) connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 84 85 85 87 88 92 Chapter 7. Open Systems Adapter-Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 7.1 Functional description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.1.1 Operating modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.1.2 QDIO mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 7.1.3 Non-QDIO mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.1.4 OSA addressing support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7.1.5 OSA/SF support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.2 OSA capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.2.1 Virtual IP address (VIPA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.2.2 Primary/secondary router function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.2.3 IPv6 support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 7.2.4 Large send for TCP/IP traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 7.2.5 VLAN support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 7.2.6 SNMP support for z/OS and Linux on System z . . . . . . . . . . . . . . . . . . . . . . . . . 106 7.2.7 TCP/IP multicast and broadcast support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.2.8 ARP cache management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.2.9 IP network availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.2.10 Checksum offload support for z/OS and Linux on System z. . . . . . . . . . . . . . . 109 7.2.11 Dynamic LAN idle for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 7.2.12 QDIO optimized latency mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 vi IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444TOC.fm 7.2.13 Layer 2 support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.14 QDIO data connection isolation for z/VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.15 QDIO interface isolation for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.16 Layer 3 VMAC for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.17 Enterprise Extender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.18 TN3270E Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.19 OSA for NCP support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.20 Adapter interruptions for QDIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.21 Inbound workload queueing (IWQ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.22 Network management - query and display OSA configuration . . . . . . . . . . . . . 7.2.23 OSA-Express3 for the zEnterprise ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 OSA features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 OSA function support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.4 Resource Measurement Facility (RMF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 111 112 113 114 114 114 116 116 118 118 118 119 127 129 129 129 130 Chapter 8. OSA-Express Console support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 132 133 134 135 135 Chapter 9. HiperSockets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 HiperSockets benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.2 HiperSockets function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Supported functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Broadcast support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 VLAN support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 IPv6 support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.4 HiperSockets Network Concentrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.5 HiperSockets Layer 2 support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.6 HiperSockets multiple write facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.7 zIIP-Assisted HiperSockets for large messages . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.8 HiperSockets Network Traffic Analyzer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 138 138 140 142 144 144 144 144 145 145 146 146 146 147 147 Chapter 10. Coupling links and common time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 System z cluster technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.1 Coupling links and STP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.2 Multi-site sysplex considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Coupling link options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.2 Internal Coupling (IC) link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.3 InterSystem Channel-3 (ISC-3). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.4 Integrated Cluster Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.5 InfiniBand coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.6 Fanout rebalancing (FC 2400) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 150 152 152 153 153 155 156 157 158 162 Contents vii 5444TOC.fm Draft Document for Review July 21, 2010 6:06 pm 10.3 Time functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 NTP server with PPS support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Sysplex Timer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 164 165 166 166 Chapter 11. Extended distance solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Unrepeated distances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Conversion to parallel channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Enterprise Systems Connection (ESCON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 ESCON unrepeated distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.2 ESCON repeated distance solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Fibre Connection (FICON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.1 FICON unrepeated distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.2 FICON repeated distance solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 Wavelength Division Multiplexing (WDM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.1 System z GDPS qualification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.2 System z qualified WDM vendor products . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 168 170 172 172 172 176 176 177 179 181 181 183 183 Appendix A. CHPID assignment and CHPID mapping tool . . . . . . . . . . . . . . . . . . . . . Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHPID Mapping Tool (CMT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CMT requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CMT purpose and description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHPID mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . InfiniBand support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 186 186 186 187 187 192 194 Appendix B. Fiber optic cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Connector types for fiber cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mode Conditioning Patch (MCP) cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . InfiniBand cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conversion kits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 196 197 198 199 200 202 Appendix C. Fiber cabling services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fiber cabling services options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Site and Facilities Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 204 204 213 214 Appendix D. Cryptographic features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Crypto Express2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Crypto Express3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 216 216 217 218 Appendix E. Channel feature attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 IBM Redbooks publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 viii IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444TOC.fm Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 How to get IBM Redbooks publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Contents ix 5444TOC.fm x IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444spec.fm Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM’s application programming interfaces. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. xi 5444spec.fm Draft Document for Review July 21, 2010 6:06 pm Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX® CICS® developerWorks® DS8000® ECKD™ Enterprise Storage Server® ESCON® FICON® GDPS® Geographically Dispersed Parallel Sysplex™ HiperSockets™ IBM TotalStorage Proven™ IBM® IMS™ NetView® OS/400® Parallel Sysplex® PR/SM™ Processor Resource/Systems Manager™ RACF® Redbooks® Redpaper™ Redbooks (logo) ® Resource Link™ S/390® Sysplex Timer® System Storage™ System z10™ System z9® System z® Tivoli® TotalStorage Proven™ TotalStorage® VTAM® WebSphere® zEnterprise™ z/Architecture® z/OS® z/VM® z/VSE™ z9® zSeries® InfiniBand Trade Association, InfiniBand, and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. SUSE, the Novell logo, and the N logo are registered trademarks of Novell, Inc. in the United States and other countries. SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. Other company, product, or service names may be trademarks or service marks of others. xii IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444pref.fm Preface This IBM® Redbooks® publication discusses the connectivity options available for use within and beyond the data center for the following IBM System z® family of mainframes: 򐂰 򐂰 򐂰 򐂰 򐂰 zEnterprise™ 196 (z196) System z10™ Enterprise Class (z10 EC) System z10 Business Class (z10 BC) System z9® Enterprise Class (z9 EC) System z9 Business Class (z9 BC) This book highlights the hardware and software components, functions, typical uses, coexistence, and relative merits of these connectivity features. This connectivity handbook will assist readers in understanding the connectivity alternatives that are available when planning and designing their data center infrastructure. The changes to this edition are based on the System z hardware announcement dated July 22, 2010. This book is intended for data center planners, IT professionals, system engineers, technical sales staff, and network planners who are involved in the planning of connectivity solutions for System z servers. The team who wrote this book This book was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center. Bill White is a Project Leader and Senior Networking and Connectivity Specialist at the International Technical Support Organization, Poughkeepsie Center. Mario Almeida is an IBM Certified Consulting IT Specialist working as an STG technical consultant in Brazil. He has more than 30 years of experience working with IBM large systems. Mario has co-authored several IBM Redbooks publications. His areas of expertise include System z hardware, Parallel Sysplex, GDPS and capacity planning. Erik Bakker is a Senior IT Specialist working for IBM Server and Technology Group in the Netherlands. During the past 24 years he has worked in various roles within IBM and with a large number of mainframe customers. For many years he worked for Global Technology Services as a systems programmer providing implementation and consultancy services at many customer sites. He currently provides pre-sales System z® technical consultancy in support of large and small System z customers. His areas of expertise include Parallel Sysplex®, z/OS® and System z. Vicente Ranieri Jr is an Executive IT Specialist at STG Advanced Technical Support (ATS) team supporting System z in Latin America. He has more than 30 years of experience working for IBM. Vicente is a member of zChampions team, a worldwide IBM team to participate in the creation of System z technical roadmap and value proposition materials. Besides co-authoring several redbooks, he has been an ITSO guest speaker since 2001, teaching the System z security update workshops worldwide. Vicente also presents in several IBM internal and external conferences. His areas of expertise include System z security, © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. xiii 5444pref.fm Draft Document for Review July 21, 2010 6:06 pm Parallel Sysplex, System z hardware and z/OS. Vicente is a member of Technology Leadership Council – Brazil, an IBM Academy of Technology Affiliate. Jin Yang is a Senior System Service Representative at the IBM Global Technical Services in Beijing, China. He joined IBM in 1999 to support and maintain System z products for clients throughout China. Jin has been working in the Technical Support Group (TSG) providing second level support to System z clients since 2009. His areas of expertise include System z hardware, Parallel Sysplex, and FICON connectivity. Thanks to the following people for their contributions to this project: Connie Beuselinck System z Product Planning, IBM Poughkeepsie Casimer DeCusatis Distinguished Engineer and Systems Hardware Architect, IBM Poughkeepsie Lou Ricci Systems Software Development, IBM Poughkeepsie Charles Shapley System z Configuration Plans and Design, IBM Poughkeepsie A special thanks to the authors of all previous versions of this IBM Redbooks publication for their efforts in creating the groundwork for this edition. Those authors include: Bernard Filhol, Per Fremstad Wolfgang Fries, Marian Gasparovic, Parwez Hamid, Brian Hatfield, Alex Herrman, Dick Jorna, Tomaz Kramar, Hubert Kordmann, Jeff Nesbitt, Iain Neville, Frank Packheiser, Ewerson Palacio, and Ken Trowell. Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbooks document dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You’ll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html Comments welcome Your comments are important to us! We want our Redbooks publications to be as helpful as possible. Send us your comments about this or other Redbooks publications in one of the following ways: 򐂰 Use the online Contact us review form found at: ibm.com/redbooks 򐂰 Send your comments in an Internet note to: xiv IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444pref.fm [email protected] 򐂰 Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400 Preface xv 5444pref.fm xvi IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444ch01.fm 1 Chapter 1. Connectivity introduction This chapter introduces the connectivity options available on the System z family of servers. The following topics are briefly discussed: 򐂰 Connectivity support 򐂰 Channel subsystem 򐂰 Supported channel types The input/output architecture, implementation, and system-relevant rules for System z servers are described in detail in subsequent chapters. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 1 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.1 Overview Input/output (I/O) channels1 are components of the System z Channel Subsystem (CSS) and z/Architecture. They provide a pipeline through which data is exchanged between servers, or between a server and external devices. z/Architecture channel connections are referred to as channel paths. The most common attachment to a System z channel is a control unit (CU) accessed via Enterprise Systems CONnection (ESCON®) or FIbre CONnection (FICON®) channel. The CU controls I/O devices such as disk and tape drives. Server-to-server communications are most commonly implemented using InfiniBand (IFB) coupling links, Inter System Channels (ISC), Integrated Cluster Bus (ICB) channels, and channel-to-channel (CTC/FCTC) connections. Internal Coupling (IC) channel, HiperSockets™, and channel-to-channel connections can be used for communications between logical partitions within a server. The Open Systems Adapter (OSA) provides direct, industry standard LAN network connectivity and communications in a multivendor networking infrastructure. Table 1-1 shows the available channel types for IBM System z; note that not all types of channels are supported by all server models. Table 1-1 Channel and CHPID types Channel type CHPID type Description ESCON CVC Conversion Channel (ESCON to Parallel BL) CBY Conversion Channel (ESCON to Parallel BY) CNC Connection Channel (ESCON Architecture) CTC Channel-to-Channel (communicates with ESCON CNC) CFP Coupling Facility Peer (connects to another CFP) ISC-3 feature in peer mode CBP Integrated Cluster Bus Coupling Facility Peer ICB-3 and ICB-4 features CIB Coupling links using InfiniBand (connects to another CIB) ICP Internal Coupling Facility Peer (connects to another ICP) Coupling Links 1 Channels have been a standard component of all s/360, S/370, S/390®, zSeries®, and System z servers. However, this book only discusses channel types supported by zEnterprise 196, System z10, and System z9 servers. 2 IBM System z Connectivity Handbook 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm Channel type CHPID type Description Open Systems Adapter OSD Queued Direct I/O mode OSE non Queued Direct I/O mode supports SNA, APPN, or HPR traffic and TCP/IP “passthru” traffic OSC OSA-Integrated Console Controller (OSA-ICC) supports TN3270E, non-SNA DFT to IPL CECs & LPARs OSN OSA for NCP supports the CDLC protocol for Network Control Program (NCP) channel-related functions provided by IBM Communication Controller for Linux (CCL) OSX OSA for intraensemble data network (IEDN) supports application data exchange between ensemble nodes OSM OSA for intranode management network (INMN) supports platform management within an ensemble node FC Fibre connection (FICON) architecture - native FICON FCVa FICON converted (FICON to ESCON) FCP Fibre Channel Protocol (full fabric attachment of Small Computer System Interface devices) IQD Internal Queued Direct I/O FICON HiperSockets a. Only supported with FICON Express LX feature As part of system planning activities, decisions are made about where to locate the equipment, how it will be operated and managed, what the business continuity requirements are for disaster recovery, tape vaulting, and so on. The types of software (operating systems and application programs) that are intended to be used must support the features and devices on the system. The following sections briefly describe the System z server connectivity options that are currently available. 1.2 Channel Subsystem (CSS) The Channel Subsystem (CSS) provides the functionality for the System z servers to communicate with input/output (I/O) devices and the network. Additional server performance demands higher I/O bandwidth, speed, and flexibility. The CSS has evolved with the increased scalability of IBM System z servers. The CSS architecture provides functionality in the form of multiple Channel Subsystems (CSSs). Multiple CSSs can be configured within the same System z (up to four on z196, z10 EC, and z9 EC; and up to two on z10 BC, and z9 BC servers. This delivers a significant increase in I/O scalability. See Chapter 2, “Channel Subsystem overview” on page 13, for more information. Chapter 1. Connectivity introduction 3 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.3 Parallel channel While parallel channels are not supported on zEnterprise 196, System z10, and System z9 servers, some devices still require parallel connections. Using a converter that converts ESCON connections to the parallel protocol enables devices with a parallel attachment to be used. Figure 1-1 illustrates how parallel devices are attached using an ESCON converter. ESCON conversion channel ESCON link CHPID type CVC/ CBY ESCON converter Parallel Parallel CU Device Parallel CU Device Parallel CU Device Figure 1-1 Parallel channel attachment through an ESCON Converter With parallel channels, I/O devices are connected using two copper cables called bus cables and tag cables. A bus cable carries information (one byte each way), and a tag cable indicates the meaning of the data on the bus cable. Control units can be chained on parallel connections (also know as daisy chaining). The architectural limit to the number of control units in a chain is 256, but the electrical signal power requirements restrict this to a practical maximum of eight control units on a channel. The converter translates the ESCON protocol to the parallel protocol. The maximum data rate of the parallel channel is up to 4.5 MBps when in streaming mode, and the maximum distance that can be achieved with a parallel channel interface is up to 122 meters (400 ft). These specifications can be limited by the connected control units and devices. See the manufacturer’s specifications for control unit and device characteristics and limitations. 4 IBM System z Connectivity Handbook 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.4 Enterprise Systems Connection (ESCON) The ESCON protocol was introduced for I/O connectivity, using fiber optic cables, including a switched topology for data traffic. Figure 1-2 shows the general structure of ESCON connectivity. System z10 Server ESCON Director ESCO N Channels 17 M Bps ESCO N Links ESCON Channels S ystem z9 Server Tape Enterprise Storage Server Figure 1-2 ESCON connectivity ESCON provides bi-directional serial bit transmission, in which the communication protocol is implemented through sequences of special control characters and through formatted frames of characters. ESCON utilizes fiber optic cables for data transmission. The ESCON link data rate is 200 megabits per second (Mbps), which results in a maximum aggregate data rate of up to 17 megabytes per second (MBps). The maximum unrepeated distance of an ESCON link using multimode fiber optic cables is 3 km (1.86 miles) when using 62.5 micron fiber. Important: The IBM ESCON Director 9032 (including all models and features) is withdrawn from marketing. There is no IBM replacement for the IBM 9032. Third party vendors may be able to provide similar functionality to that of the IBM 9032 model 5 ESCON director. These ESCON link distances can be extended using a variety of solutions. For more information see 11.3.2, “ESCON repeated distance solutions” on page 172. Chapter 1. Connectivity introduction 5 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.5 Fibre Connection (FICON) FICON provides all the strengths of ESCON, plus more. The link data rate is 4 Gbps, with an expected effective data transfer rate of up to 350 MBps (full-duplex data transfer and large sequential read/write mix). The System z servers build on this I/O architecture by offering high speed FICON connectivity (Figure 1-3). Site B Site A FICON Bridge FCV Cascaded FICON Directors FC Switch FC Switch FC FC FICON (FC) FICON CU FCP FICON CU ESCON Director FICON Bridge Card Fibre Channel Directors ESCON 17MBps SCSI Devices IBM System Storage Figure 1-3 FICON connectivity The FICON implementation enables full duplex data transfer, so data travels in both directions simultaneously, rather than the ESCON half-duplex data transfer. Also, multiple concurrent I/Os can occur on a single FICON channel, a fundamental difference between FICON and ESCON. FICON channels experience minimal data rate droop at distances up to 100 km (62 miles). This is in marked contrast to the significant droop of ESCON channels at distances beyond 9 km (5.6 miles), and is due to the fundamentally different data transfer methods of the two transmission protocols. FICON link distances can be extended using a variety of solutions. For more information see 11.4.2, “FICON repeated distance solutions” on page 177. Important: FICON Bridge (FCV) is only supported with the FICON Express LX feature, on upgrades to System z10 and System z9 servers. zEnterprise 196 servers do not support FCV channels. The FICON features on System z also support full fabric connectivity with z/VM, z/VSE™, and Linux on System z for the attachment of Small Computer System Interface (SCSI) devices using the Fibre Channel Protocol (FCP). 6 IBM System z Connectivity Handbook 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.6 Open Systems Adapter-Express The Open Systems Adapter-Express (OSA-Express) features bring the strengths of System z, such as security, availability, enterprise-wide access to data, and systems management to the client/server environment. OSA-Express2 and OSA-Express3 features are designed to provide direct, industry standard, local area network (LAN) connectivity in a multivendor networking infrastructure (Figure 1-4). Servers System z Server 10 Gigabit Ethernet (10 Gbps) Gigabit Ethernet (1 Gbps) 1000BASE-T (10/100/1000 Mbps) Ethernet Figure 1-4 OSA-Express connectivity for IBM System z OSA-Express provides connectivity support to the following LAN types: 򐂰 1000BASE-T Ethernet (10/100/1000 Mbps) 򐂰 1 Gbps Ethernet 򐂰 10 Gbps Ethernet The OSA-Express2 features (10 Gigabit Ethernet, Gigabit Ethernet (GbE), and 1000BASE-T) were available for System z10 and System z9 servers, while the OSA-Express3 features (10 Gigabit Ethernet, Gigabit Ethernet (GbE), and 1000BASE-T) are the current offerings on zEnterprise 196 and System z10 servers. OSA-Express2 features can be carried forward to a z196 from a z9 or a z10. Note that the OSA-Express2 10 GbE and GbE features have been withdrawn on z10, 30 June 2009. Chapter 1. Connectivity introduction 7 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.7 HiperSockets HiperSockets provide a seamless network to consolidate servers into an advanced infrastructure intraserver network. HiperSockets creates multiple independent, integrated, virtual LANs within System z. Up to 32 HiperSockets can be configured on zEnterprise 196 servers and up to 16 HiperSockets can be configured on System z10 and System z9 servers. HiperSockets are accessible among combinations of logical partitions or virtual servers within the System z. This network within the box concept minimizes network latency and maximizes bandwidth capabilities between z/VM, Linux on System z, z/VSE, and z/OS images, or combinations of these. It is also possible to have HiperSockets under z/VM, which permits internal networks between guest operating systems, such as many Linux servers, for example. Each HiperSockets LAN uses a Channel Path Identifier (CHPID) with a channel type IQD. Figure 1-5 shows an example of HiperSockets connectivity with multiple LPs and virtual servers. z/VM LP-1 Sysplex A Linux Guest 1 Linux Guest 2 z/VM Linux Guest n TCP/IP z/OS LP-5 HiperSockets FD Application traffic separation z/OS LP-7 z/OS LP-6 Sysplex traffic isolated from non-sysplex traffic FE Different sysplex traffic on same server FF Linux LP-2 Linux LP-3 Linux LP-4 z/OS LP-8 FC z/OS LP-9 z/OS LP-10 Sysplex B Figure 1-5 HiperSockets connectivity HiperSockets on System z is a technology that provides high-speed connectivity between virtual servers within a System z server. It eliminates the need for any physical cabling or external networking connection between these virtual servers. HiperSockets are implemented in the Licensed Internal Code (LIC) of a System z, with the communication path being within system memory. The connections between the virtual servers form a virtual LAN. HiperSockets uses internal Queued Direct Input/Output (iQDIO) at memory speed to transfer information between the virtual servers. 8 IBM System z Connectivity Handbook 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.8 Parallel Sysplex and coupling links The Parallel Sysplex® technology represents a synergy between hardware and software and is comprised of Parallel Sysplex-capable servers, Coupling Facility, coupling links (PSIFB, IC, ICB, and ISC-3), Server Time Protocol (STP), optionally Sysplex Timers, shared DASD, and software (both system and subsystem) designed for parallel processing (Figure 1-6). System z z/OS CF01 ICF PS IS System z C- IF B 3 ST ST P System z P z/OS CF02 ICF z/OS STP ICB IC Sysplex Timer 11 12 1 10 2 9 3 8 4 7 6 5 11 12 1 10 2 9 3 8 4 7 6 5 ESCON / FICON DASD DASD DASD Figure 1-6 Parallel Sysplex connectivity with mixed Coordinated Timing Network (CTN) Parallel Sysplex technology is a highly advanced, clustered, commercial processing system. It supports high-performance, multisystem, read/write data sharing, enabling the aggregate capacity of multiple z/OS systems to be applied against common workloads. The systems in a Parallel Sysplex configuration have all the capabilities of the System z and run the same applications. This allows users to harness the power of multiple System z systems as though they were a single logical computing system. The architecture is centered around the implementation of a Coupling Facility running the Coupling Facility Control Code (CFCC) and high-speed coupling connections for intersystem and intrasystem communications. The Coupling Facility is responsible for providing high-speed data sharing with data integrity across multiple System z servers. Parallel Sysplex design is implemented for high availability of business critical applications. The design is further enhanced with the introduction of System-Managed Coupling Facility Structure Duplexing, which provides additional benefits: 򐂰 Availability: Structures do not need to be rebuilt in the event of a coupling facility failure. 򐂰 Manageability and Usability: A consistent procedure is established to manage structure recovery across exploiters. 򐂰 Configuration Benefits: A sysplex can be configured with only Internal CFs. Chapter 1. Connectivity introduction 9 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm 1.9 System z channel feature support Table 1-2 lists the channel features available on System z servers. Not all channel features can be ordered on all server families. Some features are only available when carried forward on a server upgrade. Table 1-2 System z server channel feature support Max. number of channels or ports Channel feature Feature codes ESCON 16-port ESCONa (1 spare port per card) z9 EC z9 BC R07/S07 z10 EC z10 BC z196 Ports Channel per card increments (orderable) Chapter 4, “Enterprise Systems Connection” on page 37 2323 2324 (port act) FICON 1024 240/420 1024 (960 on E12) 480 240 360d 15+1b 4 (LIC-CC) Chapter 5, “Fibre Connection” on page 49 FICON Express LX 2319 120c 32/40c 120c 40c N/A 2 2 FICON Express SX 2320 120c 32/40c 120c 40c N/A 2 2 FICON Express2 LX 3319 336 64/80c 336c 112d N/A 4 4 336c 112c N/A 4 4 FICON Express2 SX 3320 336 64/80c FICON Express4 10 KM LX 3321 336 64/112 336 128 288c 336cd 4 4 FICON Express4 SX 3322 336 64/112 336 128 288c 336cd 4 4 FICON Express4-2C SX 3318 n/a 32/56 n/a 64 N/A 2 2 FICON Express4-2C 4 KM LX 3323 n/a 32/56 n/a 64 N/A 2 2 FICON Express4 4 KM LX 3324 336 64/112 336 128 288c 336cd 4 4 FICON Express8 10 KM LX 3325 N/A N/A 336 128 288 336d 4 4 FICON Express8 SX 3326 N/A N/A 336 128 288 336d 4 4 Chapter 7, “Open Systems Adapter-Express” on page 93 Chapter 8, “OSA-Express Console support” on page 131 OSA-Express 1364 48c 30/40c N/A N/A N/A 2 2 OSA-Express GbE LX 2364 24 c 24/24c N/A N/A N/A 2 2 OSA-Express GbE SX 1365 48c 30/40c N/A N/A N/A 2 2 OSA-Express GbE SX 2365 24c 24/24c N/A N/A N/A 2 2 OSA-Express 1000BASE-T Ethernet 1366 48c 30/40c N/A N/A N/A 2 2 OSA-Express Fast Ethernet 2366 24c 24/24c N/A N/A N/A 2 2 OSA-Express2 GbE LX 3364 48 30/48 48 48 48c 2 2 OSA-Express GbE LX 10 IBM System z Connectivity Handbook 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm Max. number of channels or ports Channel feature Ports Channel per card increments (orderable) Feature codes z9 EC z9 BC R07/S07 z10 EC z10 BC z196 OSA-Express2 GbE SX 3365 48 30/48 48 48 48c 2 2 OSA-Express2 1000BASE-T Ethernet 3366 48 30/48 48 48 48c 2 2 OSA-Express2 10 GbE LR 3368 24 15/24 24c 24c N/A 1 1 OSA-Express3 GbE LX 3362 N/A N/A 96 96 96 4 4 OSA-Express3 GbE SX 3363 N/A N/A 96 96 96 4 4 OSA Express3 1000BASE-T Ethernet 3367 N/A N/A 96 96 96 4 4 OSA-Express3-2P 1000BASE-T Ethernet 3369 N/A N/A N/A 48 N/A 2 2 OSA-Express3 10 GbE LR 3370 N/A N/A 48 48 48 2 2 OSA-Express3 10 GbE SR 3371 N/A N/A 48 48 48 2 2 OSA-Express3-2P GbE SX 3373 N/A N/A N/A 48 N/A 2 2 32 N/A N/A HiperSockets HiperSockets Chapter 9, “HiperSockets” on page 137 N/A Coupling links 16 16/16 16 16 Chapter 10, “Coupling links and common time” on page 149 IC N/A 32 32/32 32 32 32 N/A N/A ICB-3 (1 GBps)e 0993 16 16/16 N/A N/A N/A N/A 1 ICB-4 (2 GBps) 3393 16 16/16 16 12 N/A N/A 1 ISC-3 (2 Gbps)f 0217 (ISC-M) 0218 (ISC-D) 0219 (port act) 48 48/48 48 48 48 4 1 (LIC-CC) HCA2-O (PSIFB) 6 or 3 GBps 0163 N/A N/A 32 12 32 2 2 HCA2-O LR (PSIFB) 5 or 2.5 Gbps 0168 N/A N/A 32 12 32 2 2 HCA1-O (PSIFB) 3 GBps 0167 16 16 N/A N/A N/A 2 2 Crypto Appendix D, “Cryptographic features” on page 215 Crypto Express2 0863 16 8/16 16 16 N/A 2 2g Crypto Express2-1P 0870 N/A 4/8 N/A 8 N/A 1 1h Crypto Express3 0864 N/A N/A 16 16 16 2 2g Crypto Express3-1P 0871 N/A 8 N/A 1 1h a. The ESCON 16-port card feature code is 2323, while individual ESCON ports are ordered in increments of four using feature code 2324. Chapter 1. Connectivity introduction 11 5444ch01.fm Draft Document for Review July 21, 2010 6:06 pm b. The feature 2323 provides 15 ports that can activated via feature code 2324 plus one spare port. c. Carry forward on an upgrade d. RPQ 8P2506 is required for more than 72 I/O features. RPQ 8P2507 is required for more than 240 ESCON channels. e. RPQ 8P2197 is available for the ISC-3 Long Distance Option (up to 20 km) and has an increment of 2 without additional LIC-CC. f. There are two feature codes for the ISC-3 card: Feature code 0217 is for the ISC Mother card (ISC-M) Feature code 0218 is for the ISC Daughter card (ISC-D). Individual ISC-3 port activation must be ordered using feature code 0219 g. Two features (4 ports) initially, then one feature (2 ports) thereafter. h. Two features (2 ports) initially, then one feature (1 port) thereafter. Maximum combined features All I/O cage or drawer slots can be fully populated with these features. The number of combined features supported on z196, z10, and z9 servers is shown in Table 1-3. Table 1-3 System z maximum number of I/O features z9 EC, three I/O cages 84 features z9 BC, one I/O cage z10 EC, three I/O cages 28 features 84 features z10 BC, four I/O drawers 32 features z196, maximum I/O cages and drawers 72 featuresa a. For more than 72 I/O features, RPQ 8P2506 is required Note: For the z9 BC (R07) the limit is 16 features. The z10 EC (E12) and z9 EC (S08) support up to 64 features (with 8 HCA2-C fanout cards). The z9 BC model R07 (capacity setting A01 only) has most of the same features as the other capacity settings; however, it has smaller I/O configuration/expansion capabilities. Capacity setting A01 supports 16 slots in the I/O cage and allows for a maximum of the following: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 240 ESCON channels 32 FICON Express channels 64 FICON Express2 channels 32 FICON Express4 channels (2-port features) 64 FICON Express4 channels (4-port features) 24 OSA-Express ports 24 OSA-Express2 ports (12 ports with 10 GbE LR features) 16 ICB-3 links 8 ICB-4 links 48 ISC-3 links 8 Crypto Express2 adapters (2-adapter features) 4 Crypto Express2-1P adapters (1-adapter features) These numbers are maximum numbers per feature. The total number of combined features that can be installed is limited by the number of available slots. 12 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch02.fm 2 Chapter 2. Channel Subsystem overview This chapter describes the Channel Subsystem (CSS). The CSS handles all the system input/output (I/O) operations for System z servers. Its role is to control communication between internal or external channels, and control units and devices. The following topics are discussed: 򐂰 򐂰 򐂰 򐂰 CSS overview Multiple CSS concept I/O configuration management I/O configuration planning © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 13 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm 2.1 CSS description A CSS enables communication from server memory to peripherals via channel connections. The channels in the CSS permit transfer of data between main storage and I/O devices or other servers under the control of a channel program. The CSS allows channel I/O operations to continue independently of other operations within the server. This allows other functions to resume after an I/O operation has been initiated. The CSS also provides communication between logical partitions within a physical server using internal channels. 2.1.1 CSS elements A CSS can have up to 256 channel paths. A channel path is a single interface between a server and one or more control units. Commands and data are sent across a channel path to perform I/O requests. The entities that encompass the CSS are described in this section. Subchannels A subchannel provides the logical representation of a device to the program and contains the information required for sustaining a single I/O operation. A subchannel is assigned for each device defined to the logical partition. Three subchannel sets available to z196, while two subchannel sets are on z10 and z9. Channel Path Identifier A Channel Path Identifier (CHPID) is a value assigned to each channel path of the system that uniquely identifies that path. A total of 256 CHPIDs are supported by a CSS. The channel subsystem communicates with I/O devices by means of channel paths between the channel subsystem and control units. On System z a CHPID number is assigned to a physical location (slot/port) by the user via HCD or IOCP. Control units A control unit provides the logical capabilities necessary to operate and control an I/O device and adapts the characteristics of each device so that it can respond to the standard form of control provided by the CSS. A control unit may be housed separately, or it may be physically and logically integrated with the I/O device, the channel subsystem, or within the server itself. I/O devices An input/output (I/O) device provides external storage, a means of communication between data-processing systems, or a means of communication between a system and its environment. In the simplest case, an I/O device is attached to one control unit and is accessible through one channel path. 2.1.2 Multiple CSS concept The multiple Channel Subsystem (CSS) concept is implemented in the System z servers. The z196, z10 EC, and z9 EC support up to four CSSs, while the z10 BC and z9 BC support up to two CSSs. The design of the System z servers offers considerable processing power, memory sizes, and I/O connectivity. In support of the larger I/O capability, the CSS concept has been scaled up correspondingly. This provides relief for the number of supported Logical Partitions, channels, and devices available to the server. 14 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm Each CSS can have from 1 to 256 channels and can in turn be configured with 1 to 15 logical partitions, with a maximum of 30 logical partitions per server on z10 BC and z9 BC, and up to 60 logical partitions on z196, z10 EC, and z9 EC servers. CSSs are numbered from 0 to 3, the numbers sometimes referred to as the CSS Image ID (CSSID 0, 1, 2, or 3 for the z196, z10 EC, and z9 EC; and CSSID 0 and 1 for the z10 BC and z9 BC). 2.1.3 Multiple CSS structure The structure of multiple CSSs provides channel connectivity to the defined logical partitions in a manner that is transparent to subsystems and application programs. The System z servers provide the ability to define more than 256 CHPIDs in the system through the multiple CSSs. As mentioned, CSS defines CHPIDs, control units, subchannels, and so on. This enables the definition of a balanced configuration for the processor and I/O capabilities. For ease of management, it is strongly recommended that the Hardware Configuration Definitions (HCD) be used to build and control the System z input/output configuration definitions. HCD support for multiple Channel Subsystems is available with z/VM and z/OS. HCD provides the capability to make both dynamic hardware and software I/O configuration changes. Logical Partitions cannot be added until at least one CSS has been defined. Logical Partitions are defined to a CSS, not to a server. A Logical Partition is associated with one CSS only. CHPID numbers are unique within a CSS; however, the same CHPID number can be reused within all CSSs. Important: The System z servers do not support basic mode. Only logically partitioned (LPAR) mode can be defined. All Channel Subsystems are defined within a single I/O Configuration Data Set (IOCDS). The IOCDS is loaded into the Hardware System Area (HSA) and initialized during Power-On Reset. On z196 and z10 EC servers the HSA has a fixed size of 16 GB and on the z10 BC server the HSA has a fixed size of 8 GB, which is not included in the purchased memory. On System z9 the HSA allocation is controlled via the maximum number of devices field on the HCD Channel Subsystem List panel. On System z9 this value can only be changed in HSA by a Power-On Reset. Figure 2-1 shows a logical view of these relationships. Note that each CSS supports up to 15 logical partitions. Logical Partitions CSS 3 CSS 2 CSS 0 CSS 1 up to 256 up to 256 up to 256 up to 256 CHPIDs CHPIDs CHPIDs CHPIDs HSA IOCDS Up to 15 Logical Partitions per CSS Max. 30 Logical Partitions Max. 60 Logical Partitions (z196, z10 EC, z9 EC only) Up to 4 Channel Subsystems Max. of 2 CSSs on z9 BC and z10 BC Single HSA with one active IOCDS that provides a single system image for I/O Interconnect cables Physical Channels (PCHIDs) I/O Cage with features Figure 2-1 Logical view of CSSs, IOCDS, and HSA Chapter 2. Channel Subsystem overview 15 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm 2.1.4 Multiple Image Facility (MIF) MIF enables resource sharing across logical partitions within a single CSS or across CSSs. When a channel is shared across logical partitions in different CSSs, it is called “spanning.” See 2.1.8, “Channel spanning” on page 21 for more information. With multiple CSSs, the IOCDS logical partition MIF Image ID is not unique. Therefore, the logical partition identifier value provides a unique value for each logical partition within the same server. The terminology discussed in the following sections applies. Logical partition identifier The logical partition identifier is a number in the range of 00 to 3F and is assigned in the image profile through the Support Element (SE) or the Hardware Management Console (HMC). It is unique across the server and can also be referred to as the User Logical Partition ID (UPID). Logical partition name The logical partition name is defined through HCD or IOCP. It is the name in the RESOURCE statement in the IOCP. The logical partition names must be unique across all CSSs. MIF ID This identifier is defined through HCD or IOCP. This is the number defined in the RESOURCE statement in the IOCP. It is in the range of 1 to F and is unique within a CSS. It is not unique across multiple CSSs. Multiple CSSs can specify the same MIF ID (also known as Image ID (IID)). It is suggested to establish a numbering convention for the logical partition identifiers. As shown in Figure 2-2, use the CSS number concatenated to the MIF Image ID, which means logical partition ID 3A is in CSS 3 with MIF ID A. This fits within the allowed range of logical partition IDs and conveys useful information to the user. CSS0 CSS ID is specified in HCD / IOCP CSS3 CSS2 CSS1 TST1 PROD1 PROD2 TST2 PROD3 PROD4 TST3 TST4 PROD5 02 04 0A 14 16 1D 22 35 3A 2 4 A 4 6 D 2 5 A Logical Partition Name is specified in HCD / IOCP Logical Partition ID is specified in HMC Image Profile MIF ID is specified in HCD / IOCP Figure 2-2 Logical partition and MIF definitions Dynamic addition or deletion of a logical partition name In order to have a partition defined for future use, it must be reserved with a CSS beforehand in the IOCDS that is used for Power-On Reset. A reserved partition is defined with a partition name placeholder, a MIF ID, a usage type, and a description. The reserved partition can be assigned a logical partition name to be later used in I/O commands of HCD. Note that on a 16 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm z10 all possible CSSs and LPs are reserved automatically; the user does not need to reserve them. 2.1.5 Physical Channel ID (PCHID) A Physical Channel ID, or PCHID, reflects the physical location of a channel-type interface. A PCHID number is based on the I/O cage location, the channel feature slot number, and the port number of the channel feature. A CHPID does not directly correspond to a hardware channel port, but is assigned to a PCHID in HCD or IOCP. PCHIDs identify the physical ports on cards located in I/O cages or I/O drawers. The PCHID numbering scheme of I/O cages shown in Table 2-1. Table 2-1 PCHIDs numbering scheme Cage Front PCHID ## Rear PCHID ## I/O Cage 1a 100 - 1FF 200 - 2BF I/O Cage 2 300 - 3FF 400 - 4BF I/O Cage 3b 500 - 5FF 600 - 6BF CPC Cagec 000 - 0FF reserved for ICB-4sd a. Only I/O cage 1 is present in a z9 BC server, whereas z196, z10 EC, and z9 EC can have up to three. b. An RPQ is required for a third I/O cage in a z196 c. Although the z9 BC server has only one processor book, the reserved addresses for ICB-4s will still fit in the specified range. d. Not available on z196. z196 supports combination of I/O drawers and I/O cages, up to 4 I/O drawers can be installed, the z196 I/O drawer numbering scheme is shown in Table 2-2. Table 2-2 z196 I/O drawer PCHID numbering scheme Drawer Front PCHID ## Rear PCHID ## I/O drawer 1 580-5BF 5C0-5FF I/O drawer 2 500-53F 540-57F I/O drawer 3 380-3BF 3C0-3FF I/O drawer 4 300-33F 340-37F On a z10 BC PCHIDs identify the physical ports on cards located in I/O drawers. The numbering scheme is shown in Table 2-3. Table 2-3 z10 BC PCHID numbering scheme Drawer Front PCHID ## Rear PCHID ## I/O drawer 3 280-2BF 2C0-2FF I/O drawer 2 200-23F 240-27F I/O drawer 1 180-1BF 1C0-1FF I/O drawer 4 100-13F 140-17F CPC drawer 000-00B reserved for ICB-4 Chapter 2. Channel Subsystem overview 17 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm CHPIDs are not pre-assigned on System z servers. It is the user’s responsibility to assign the CHPID numbers through the use of the CHPID Mapping Tool (CMT) or directly with HCD/IOCP. Assigning CHPIDs means that the CHPID number is associated with a physical channel port location (PCHID) or the Adapter ID (AID), and a CSS. The CHPID number range is still from 00 to FF and must be unique within a CSS. Any CHPID not connected to a PCHIDs will fail validation when an attempt is made to build a production IODF or an IOCDS. Figure 2-3 shows the front view of the first I/O cage (bottom of the A frame), including some I/O cards in slots 01 to 08, and the PCHID numbers of each port. I/O Feature Crypto ISC-3 OSA-E OSA-E STI FICON FICON 140 150 110 120 130 I/O Ports ESCON 160 161 162 163 141 151 111 PCHIDs ... STI 142 152 16B 121 131 143 153 I/O Slots 01 02 03 04 05 06 07 08 ... Figure 2-3 PCHID example: front of I/O cage1 Example 2-1 contains the corresponding PCHID report of the configuration shown in Figure 2-3 on page 18. A z9 EC PCHID report is used in the example. Example 2-1 Physical Channel ID (PCHID): report example for a z9 EC server Machine: 2094-S18 2991E - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Book/Fanout/Jack Cage Slot F/C PCHID/Ports Comment 0/D8/J00 A19B 06 3393 01E/J00 1/D8/J00 A19B 10 3393 1/D7/J00 A01B 01 0864 100/P00 101/P01 0/D7/J00 A01B D102 0218 110/J00 111/J01 1/D7/J00 A01B 03 3365 120/J00 121/J01 0/D7/J00 A01B 04 3365 130/J00 131/J01 1/D7/J00 A01B 06 3319 140/J00 141/J01 142/J02 143/J03 0/D7/J00 A01B 07 3319 150/J00 151/J01 152/J02 153/J03 1/D7/J00 A01B 08 2323 160/J00 161/J01 162/J02 163/J03 164/J04 165/J05 166/J06 167/J07 168/J08 169/J09 16A/J10 16B/J11 Legend: 18 IBM System z Connectivity Handbook 02E/J00 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm A19B A01B D1xx 3365 3319 2323 0864 0218 3393 Top of A frame Bottom of A frame Half high card in top of slot xx OSA Express2 GbE SX (2 port) FICON Express2 LX (4 port) ESCON Channel 16 Ports Crypto Express3 ISC D <10KM ICB 4 Function Enablement From this PCHID report, you can observe the following: 򐂰 I/O slot 02 has an ISC-3 Daughter (ISC-D) half-high card (FC 0218) in the top, connected to fanout D7 (Jack J.00) of book 0. Its two enabled ports have PCHID numbers 110 and 111. 򐂰 I/O slot 03 and 04 have a OSA Express2 GbE SX card; card in slot 03 is connected to fanout D7 of book 1 (PCHIDs 120/121) and card in slot 04 is connected to fanout D7 of book 0 (PCHIDs 130/131). 򐂰 The FICON Express2 LX cards in slot 06 and 07 are connected to fanout D7 of book 1 (slot 06) and to fanout D7 of book 0 (slot 07), and the 4 ports in slot 06 have PCHID numbers 140, 141, 142, and 143. 򐂰 A 16 port ESCON card is installed in slot 08, connected to fanout D7 of book 1, and has 12 ports enabled, PCHIDs 160 to 16B. A PCHID report can be obtained from the IBM account team when an order is place or proposed. 2.1.6 Adapter ID (AID) The AID number is used to assign a CHPID to a port via HCD/IOCP for Parallel Sysplex over InfiniBand (PSIFB) coupling links. The AID is a number between 00 and 1F on z196, z10 EC, and z9 EC, or 08-0F on a z9 BC. On the z10 BC the AIDs are numbered from 00 to 05. The AID for the z9 is bound to the physical fanout position. If the fanout is moved to another slot, the AID changes for that specific fanout and it might be necessary to readjust the IOCDS. For z196 and z10, the AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. No IOCDS update is required if adapters are moved to a new physical location. Table 2-4 shows the assigned AID numbers for a new build z196 or z10 EC. Table 2-4 Fanout AID numbers for z196 or z10 EC Fanout location Book Fourth First Third Second D1 00 08 10 18 D2 01 09 11 19 D3 N/A N/A N/A N/A D4 N/A N/A N/A N/A Chapter 2. Channel Subsystem overview 19 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm Fanout location Book D5 02 0A 12 1A D6 03 0B 13 1B D7 04 0C 14 1C D8 05 0D 15 1D D9 06 0E 16 1E DA 07 0F 17 1F Table 2-5 shows the assigned AID numbers for a new build z10 BC. Table 2-5 Fanout AID numbers for z10 BC Fanout location AID D3 00 D4 01 D5 02 D6 03 D7 04 D8 05 The AIDs are shown in the PCHID report provided by an IBM representative for new build z196 servers or for upgrades. Part of a PCHID report is shown in Example 2-2. Example 2-2 AID assignment in z196 PCHID report CHPIDSTART 19756694 PCHID Machine: 2817-M32 SNxxxxxxx - - - - - - - - - - - - - - - - - Source Cage Slot F/C 06/D6 A25B D606 0163 15/D6 A25B D615 0163 REPORT - - - - - - - - - - - - - - - - - - - - PCHID/Ports or AID Comment AID=0B AID=1B For more information regarding PSIFB coupling link features, refer to 10.2.5, “InfiniBand coupling links” on page 158. 2.1.7 Multiple CSS construct Figure 2-4 shows a pictorial view of 2 CSSs defined as CSS0 and CSS1. Each CSS has three Logical Partitions with their associated MIF Image Identifiers. In each CSS, the CHPIDs are shared across all Logical Partitions. The CHPIDs are assigned to their designated PCHIDs. 20 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm CSSs CSS 0 CSS 1 LP Names LP1 LP2 LP3 LP14 LP15 LP16 MIF IDs 1 2 3 1 2 3 CHPIDs 80 81 90 91 80 81 90 91 PCHIDs 140 150 1E0 1F0 141 151 1E1 1F1 Directors Control Units and Devices DASD LCUs DASD LCUs Figure 2-4 Multiple CSS construct The CHPIDs are mapped to their designated PCHIDs using the CHPID Mapping Tool (CMT), or manually, using HCD or IOCP. The output of CMT is used as input to HCD or the IOCP to establish the CHPID to PCHID assignments. See 2.2, “I/O configuration management” on page 24, for further details regarding the CMT. 2.1.8 Channel spanning Channel spanning extends the MIF concept of sharing channels across logical partitions to sharing channels across Channel Subsystems and logical partitions. Spanning is the ability for the channel to be configured to multiple Channel Subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the Channel Subsystem to which the logical partition is configured. A channel is considered a spanned channel if the same CHPID number in different CSSs is assigned to the same PCHID in IOCP, or is defined as “spanned” in HCD. In the case of internal channels (for example, IC links and HiperSockets), the same applies, but there is no PCHID association. They are defined with the same CHPID number in multiple CSSs. CHPIDs that span CSSs reduce the total number of channels available on System z. The total is reduced since no CSS can have more than 256 CHPIDs. For a z10 BC, or z9 BC that supports two CSSs, a total of 512 CHPIDs are supported. If all CHPIDs are spanned across the two CSSs, then only 256 channels can be supported. For a z196, or z10 EC, or z9 BC Chapter 2. Channel Subsystem overview 21 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm with four CSSs, a total of 1024 CHPIDs are supported. If all CHPIDs are spanned across the four CSSs, then only 256 channels can be supported. Table 2-6 shows which channel types can be defined as shared or spanned channels. Table 2-6 Spanned and shared channels Channel type ESCON FICON Express External External CHPID definition Shared channels Spanned channels CNC, CTC Yes No CVC, CBY No No FC, FCP Yes Yes FCV Yes No FICON Express2 External FC, FCP Yes Yes FICON Express4 External FC, FCP Yes Yes FICON Express8 External FC, FCP Yes Yes OSA-Expressa External OSD, OSE, OSC Yes Yes OSA-Express2b External OSD, OSE, OSC, OSN Yes Yes OSA Express3c External OSD, OSE, OSC, OSN, OSXd, OSMe Yes Yes PSIFB External CIB Yes Yes ICB-4 External CBP Yes Yes ICB-3 External CBP Yes Yes ISC-3 External CFP Yes Yes IC Internal ICP Yes Yes HiperSockets Internal IQD Yes Yes a. OSA Express GbE features only support OSD. b. OSA-Express2 GbE features only support OSD and OSN, OSA-Express2 10 GbE feature only support OSD c. OSA-Express3 GbE features only support OSD and OSN, OSA-Express3 10 GbE feature only support OSD and OSX d. Only OSA Express3 10 GbE features can be defined as CHPID type OSX e. Only OSA Express3 1000BASE-T feature can be defined as CHPID type OSM Note: Spanning of ESCON channels and FICON converter (FCV) channels is not supported. In Figure 2-5, CHPID 04 is spanned across CSS 0 and CSS 1. Since it is not an external channel link there is no PCHID assigned. CHPID 06 is a spanned external channel and has a PCHID 120 assigned. 22 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm Partition Partition 1 Partition Partition Partition Partition Partition 2 14 15 16 17 30 CSS0 MIF-1 MIF-2 CSS1 MIF-F CHPID 03 Share CHPID FF PCHID PCHID PCHID PCHID 10B 10C 10D 20E PCHID 20A CHPID CHPID CHPID 00 01 02 Partition 18 Internal MIF-1 CHPID 04 SPAN MIF-3 CHPID CHPID 00 01 CHPID 06 SPAN PCHID 120 MIF-2 MIF-F CHPID CHPID CHPID 05 22 FF Share PCHID PCHID PCHID PCHID PCHID 145 146 158 159 147 External Figure 2-5 Two channel subsystems with spanned channels (internal and external) 2.1.9 Multiple subchannel sets The multiple subchannel sets (MSS) functionality should not be confused with multiple channel subsystems. In most cases a subchannel represents an addressable device. For example, a disk control unit with 30 drives uses 30 subchannels. An addressable device is associated with a device number. Subchannel numbers (including their implied path information to a device) are limited to four hexadecimal digits by hardware and software architectures. Four hexadecimal digits provide 64 K addresses, known as a set. IBM reserved 256 subchannels, leaving 63.75 K subchannels for general use with the System z servers. Parallel Access to Volume (PAV) has made this limitation of subchannels a challenge for larger installations. A single disk drive (with PAV) often consumes at least four subchannels.1 Because the use of four hexadecimal digits for subchannels (and device numbers corresponding to subchannels) is architected in a number of places, it was difficult to remove this constraint. Simply expanding the field would break too many programs. The solution allows sets of subchannels (“addresses”), with a current implementation of three sets. Each set provides 64 K addresses. Subchannel set 0, the first set, reserves 256 subchannels for IBM use. Subchannel set 1 provides a full range of 64 K subchannels on z196, z10 and z9 servers. Subchannel set 2 provides another full range of 64 K subchannels on z196 servers only. In principle, subchannels in either set could be used for any device addressing purpose. However, the current implementation (in z/OS) restricts subchannel sets 1 and 2 to disk alias subchannels only. Subchannel set 0 can be used for base addresses and for alias addresses. 1 Four is a widely used number for PAV. It represents the base address and three alias addresses. Chapter 2. Channel Subsystem overview 23 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm There is no required correspondence between addresses in the three sets. For example, it is possible to have device number 8000 in subchannel set 0 and device number 8000 in subchannel sets 1 or 2 and they might refer to completely separate devices. (It is known that the device number in subchannel sets 1 and 2 must be an alias for z/OS, but that is all that is known from the device number.) Likewise, device number 1234 (subchannel set 0) and device number 4321 (subchannel set 1) might be the base and an alias for the same device. There is no required correspondence between the device numbers used in the three subchannel sets. Note that each channel subsystem (CSS) can have multiple subchannel sets, as outlined in Figure 2-6. LP1 LP2 LP3 LPA LPB CSS 0 subch set 0 63.75K dev LPC CSS 1 subch set 1 subch set 2 64K devices 64K devices subch set 0 63.75K dev subch set 1 subch set 2 64K devices 64K devices Figure 2-6 Multiple channel subsystems (CSS) and multiple subchannel sets The appropriate subchannel set number must be included in IOCP definitions (or in the HCD definitions that produce the IOCDS). The subchannel set number defaults to zero and IOCP changes are needed only when using subchannel sets 1 or 2. 2.1.10 Summary Table 2-7 shows the maximum number of CSS elements supported per System z server. Table 2-7 CSS elements z9 BC z9 EC z10 BC z10 EC z196 CSSs 2 per server 4 per server 2 per server 4 per server 4 per server Partitions 15 per CSS, up to 30a per server 15 per CSS, up to 60 per server 15 per CSS, up to 30 per server 15 per CSS, up to 60 per server 15 per CSS, up to 60 per server CHPIDs 256 per CSS, up to 512 per server 256 per CSS, up to 1024 per server 256 per CSS, up to 512 per server 256 per CSS, up to 1024 per server 256 per CSS, up to 1024 per server Devices 63.75K per CSSb, up to 127.5K per server 63.75K per CSSb, up to 255K per server 63.75K per CSSb, up to 255K per server 63.75K per CSSb, up to 255K per server 63.75K per CSSb, up to 255K per server a. z9 BC model R07 (capacity setting A01 only) supports a maximum of 15 logical partitions per server. b. Multiple subchannel sets are supported. 2.2 I/O configuration management The CSS controls communication between a configured channel, the control unit and device. The IOCDS defines the channels, control units and devices to the designated logical 24 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch02.fm partitions within the server. This is defined using the Input/Output Configuration Program (IOCP). The IOCP statements are typically built using the HCD. An interactive dialog is used to generate the IODF, invoke the IOCP program, and subsequently build the production IOCDS. The IOCDS is loaded into the Hardware System Area (HSA) and initialized during Power-On Reset. The HSA allocation is controlled via the Hardware Management Console (HMC). HSA Storage is allocated based on the size of the IOCDS (partitions, channels, control units, and devices). Additional storage is reserved for Dynamic I/O reconfiguration if enabled. On a z9 EC or a z9 BC the amount of additional storage for HSA is determined by the “maximum number of devices” field on the HCD Channel Subsystem list panel. For z9 EC and z9 BC there is a workstation-based tool for HSA estimation available from Resource Link™. The HSA on z196 and z10 EC has a fixed size of 16 GB, and on the z10 BC there is a fixed size HSA of 8 GB, not included in purchased memory. The 8 GB or 16 GB HSA is allocated during Power-On Reset. It has sufficient reserved space to allow for dynamic I/O reconfiguration changes to the maximum capability of the respective server. With System z servers, channel path identifiers are mapped to Physical Channel Identifiers (PCHID) or Adapter IDs (AID) via the configuration build process through HCD or IOCP. See 2.1.5, “Physical Channel ID (PCHID)” on page 17, and 2.1.6, “Adapter ID (AID)” on page 19. For internal channels (IC links and HiperSockets), there is no PCHID association. The following tools are provided to maintain and optimize the I/O configuration of System z servers. 2.2.1 IBM Configurator (eConfig) The eConfig tool is available to the IBM representative. It is used to build new configurations or upgrades of existing configurations, and maintains installed features of those configurations. It also produces a PCHID report. 2.2.2 Hardware Configuration Definition (HCD) HCD supplies an interactive dialog to generate the I/O definition file (IODF) and subsequently the Input/Output Configuration Dataset (IOCDS). It is strongly recommended that HCD be used to generate the IOCDS as opposed to writing IOCP statements. The validation checking that HCD performs as data is entered helps eliminate errors before the I/O configuration is implemented. 2.2.3 CHPID Mapping Tool (CMT) The CHPID Mapping Tool provides a mechanism to help with the requirements of System z, providing a mechanism to map CHPIDS to PCHIDS, and the best availability recommendations for installed features and defined configuration. It is recommended that the CMT be used for all new builds of System z, or when upgrading from one type of server to the next. It can also be used as part of standard hardware changes (such as an MES) of an existing System z. The CMT takes input from two sources: 򐂰 The Configuration Report file (CFreport) produced by the IBM order tool (eConfig) and provided by the IBM account team, or produced by IBM manufacturing and obtained from IBM Resource Link. 򐂰 An IOCP statement file. Chapter 2. Channel Subsystem overview 25 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm The following output is produced by the CMT: 򐂰 Tailored reports. All reports should be saved for reference. The Port Report sorted by CHPID number and location should be supplied to the IBM hardware service representative for System z installations. 򐂰 An IOCP input file with PCHIDs mapped to CHPIDs. This IOCP input file can then be migrated back into HCD, from which a production IODF can be built. Note that the CMT will not automatically map CIB CHPIDs to AIDs and ports. This must be done manually, either in HCD, IOCP, or the CMT. The CMT will provide availability intersects for completely defined CIB CHPIDs. For further details about the CMT, refer to Appendix A, “CHPID assignment and CHPID mapping tool” on page 185. 2.3 I/O configuration planning I/O configuration planning for System z requires the availability of physical resources and must comply to the specific rules of the logical definitions. The following physical resources are the minimum required for connectivity: 򐂰 򐂰 򐂰 򐂰 򐂰 Server frame I/O cage, I/O drawer in a server frame I/O slot in a server I/O cage or I/O drawer Channel feature in a server I/O cage or I/O drawer Port on a channel feature The System z configurator build process includes, for a server configuration, all physical resources required for a given I/O configuration, based on the supplied channel type and quantity requirements. Once the physical resources are ordered, the definition phase starts. The channels must be defined according to the architecture rules and the server’s implementation, and the actual order. There are also other definition implications imposed by the server implementation, such as the maximum number of subchannels in the Hardware System Area (HSA), for example, 3832.5K subchannels (63.75K per partition * 30 partitions + 64 K per partition * 30 partitions) in total for the z9 BC, 7665 k subchannels (63.75 K per partition * 60 partitions + 64K per partition * 60 partitions) in total for the z10 EC and z9 EC server, and 11505 k subchannels (63.75 K per partition * 60 partitions + 2 x 64K per partition * 60 partitions) in total for the z196. This will have an impact on the amount of memory used for the HSA System z server, except for z196, z10 EC, and z10 BC, which never uses user memory for HSA. The channel operational characteristics are also very important when deciding what type of channel to use. As an example, Table 2-8 shows some differences between ESCON and FICON channels. 26 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm Table 2-8 ESCON and FICON characteristics Feature ESCON channels FICON channels Command execution Synchronous to CU Asynchronous to CU Channel concurrent I/O operations 1 Up to 32 Up to 64a Link data rate 200 Mbps Up to 4 Gbpsb Channel frame transfer buffers 1 KBytes 128 KBytes Max distance without repeat 3 km 4 km and 10 km at 4 Gbps 12 km at 2 Gbps RPQ 20 km at 1 Gbps RPQ Max distance without droop 9 km 100 km a. With FICON Express2 and FICON Express4 features. b. FICON Express4 features support up to 4 Gbps, while FICON Express2 and FICON Express features support up to 2 Gbps. The operational characteristics of a given channel type, along with the addressing capabilities, can impact configuration complexity, topology, infrastructure, and performance. I/O configuration rules The following sections briefly describe the System z configuration rules identified through the architecture and the specific server, which are implemented and enforced using the Hardware Configuration Definition (HCD) and Input Output Configuration Program (IOCP). All control units and I/O devices that attach to the server must be defined to a channel subsystem (CSS). When defining the I/O configuration for the server via HCD/IOCP, specify: 򐂰 Logical partitions (logical partition name, CSS ID, and MIF ID where appropriate) 򐂰 Channel paths on the server, their assignment to CSSs and logical partitions 򐂰 ESCON or FICON Director (where appropriate) 򐂰 Control units attached to the channel paths 򐂰 I/O devices assigned to the control units Some of the given definition characteristics for the I/O configuration that must be considered are found in the architecture (z/Architecture), while others are specified just for the server itself. Chapter 2. Channel Subsystem overview 27 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm Some of the general IOCP rules for System z are shown in Table 2-9. Table 2-9 System z server attributes Constant machine attributes Maximum configurable Physical Control Units (PCUs) z196 z10 z9 PCUs per CVC, CBY 48 48 48 PCUs per OSD (OSA-Express2 and OSA Express3) 16 16 16 PCUs per OSD (OSA-Express) N/A 1 1 PCUs per OSE, OSC, OSN 1 1 1 PCUs per OSM, OSX (OSA-Express3) 16 16 16 PCUs per CBS N/A N/A N/A 1 1 1 PCUs or link addresses per CNC, CTC 120 120 120 PCUs or link addresses per FCVab, FC 256 256 256 PCUs per FCP 1 1 1 CUs per IQD 64 64 64 Per CBS N/A N/A N/A 7 7 7 Per CNC 1024 1024 1024 Per CTC 512 512 512 Per OSCc 253 253 253 Per OSD (OSA-Express2 and OSA-Express3) 1920 1920 1920 Per OSD (OSA-Epress) N/A 480 480 Per OSE 254 254 254 Per OSM, OSX (OSA-Express3) 1920 N/A N/A Per OSN 480 480 480 Per FCP 480 480 480 Per FCVa , FC 16 K 16 K 16 K For all IQD channel paths 12 K 12 K 12 K PCUs per CFP, CBPa, ICP Maximum configurable Devices Per CFP, CBPa , ICP, CIB a. Not supported on z196 b. FCV channel paths can have link addresses in the range 01 - FE and therefore have a maximum of 254 link addresses c. A limit of 120 sessions can be defined at the HMC/SE For an explanation of the CHPID types and more detail regarding channel configuration rules, refer to Input/Output Configuration Program User’s Guide, SB10-7037. 28 IBM System z Connectivity Handbook 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm System z specifications for the channel subsystem Table 2-10 summarizes the available channel CHPID types on System z and the maximum number of channels for each channel type. Table 2-10 System z maximum number of supported channel paths CHPID type Maximum number of channel paths z196 z10 EC z10 BC z9 EC z9 BC Channel paths (total) 1024a 1024b 480 1024b 420 CVC, CBY, CNC, CTC 240a 1024b 480 1024b 420 FCVc, FC, FCP 288a 336d 128 336d 112d CFP, CBPc, ICP, CIB (combined) 128 64 64 64 64 CFP 48 48 48 48 48 CBP N/A 16 12 16 16 CIB 128 64 64 64 64 ICP 32 32 32 32 32 OSD, OSE, OSC, OSN 48e 48e 48e 48 48 OSX, OSM 48 N/A N/A N/A N/A IQD 32 16 16 16 16 a. RPQ 8P2506 is required for more than 72 I/O features. RPQ 8P2507 is required for more than 240 ESCON channels. b. A z10 EC model E12 and a z9 EC model S08 support a total of 960 channel paths only. c. Not supported on z196. d. The maximum number of channel paths for FICON Express features is 120 for z10 EC and z9 EC, and 40 for z10 BC, and z9 BC. e. The maximum number of ports is 96. Note that the maximum numbers for the FICON and OSA-Express channels includes Cryptographic features installed in the System z I/O cages. Each cryptographic feature installed reduces the maximum of FCV, FC, FCP, OSD, OSE, OSC, OSN, OSX, OSM channels by one, two, or four. However, cryptographic features are not defined to HCD/IOCP, therefore HCD/IOCP cannot include them in its checking of the maximum number of channels for those types. The cryptographic features on System z do not require a CHPID definition, and are configured via the HMC or SE. 2.4 References The following publications contain information related to the topics covered in this chapter: 򐂰 z/OS Hardware Configuration Definition: User’s Guide, SC33-7988 򐂰 z/OS Hardware Configuration Definition Planning, GA22-7525 򐂰 Hardware Configuration Definition: User’s Guide, SC28-1848 򐂰 Hardware Configuration Definition Planning, GC28-1750 򐂰 Input/Output Configuration Program User’s Guide, SB10-7037 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 Business Class Technical Introduction, SG24-7241 Chapter 2. Channel Subsystem overview 29 5444ch02.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Enterprise Class Technical Introduction, SG24-7515 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Enterprise Class Technical Introduction, SG24-1515 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 30 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch03.fm 3 Chapter 3. Parallel channel This chapter provides information about the IBM System/360 and System/370 I/O interface, also called the S/360, S/370 Original Equipment Manufacturer’s Information (OEMI) interface, parallel I/O interface, or bus and tag channel interface. The following topics are covered: 򐂰 ESCON to parallel conversion 򐂰 Parallel channel modes 򐂰 Parallel channel elements Support for parallel channels for System z servers is only provided by the use of converters that change the incoming protocol to the protocol as described for parallel channels in this chapter. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 31 5444ch03.fm Draft Document for Review July 21, 2010 6:06 pm 3.1 Overview On System z servers a parallel I/O interface in a control unit can only communicate through an ESCON channel via a parallel converter unit such as the IBM 9034 converter or the Optica FXBT 34600 ESCON to B/T Converter. The purpose of the converter is to: 򐂰 򐂰 򐂰 򐂰 Send channel commands from the processor to a CU Manage the transfer data during read and write operations Receive status at the end of operations Obtain sense information from control units when required Refer to IBM System/360 and System/370 I/O Interface Channel to Control Unit Original Equipment Manufacturer’s Information, GA22-6974, for more details about parallel I/O interface architecture. 3.2 ESCON conversion to parallel channel ESCON-to-parallel channel converters are used for: 򐂰 Connecting parallel channel attached I/O control units and devices over extended distances 򐂰 Connecting parallel channel attached I/O control units and devices to zEnterprise or System z servers 򐂰 Investment protection in a parallel channel environment The IBM 9034 ESCON Converter (ESCC) model 1, and the 34600 FXBT ESCON Converter from Optica Technologies Incorporated, allow parallel channel attached I/O control units to connect to an ESCON channel. Both products convert the ESCON channel’s optical signals to parallel channel electrical signals, and the control unit’s parallel channel electrical signals to ESCON channel optical signals. System z servers support ESCON converter connections via the ESCON conversion channel CHPID types CVC and CBY. Note: The IBM 9034 ESCON Converter model 1 is withdrawn from marketing. The 34600 FXBT ESCON converter is functionally compatible with the IBM 9034. For more information, including supported parallel channel I/O control units and devices, see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by other vendors. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. Each ESCON converter, which attaches one channel to up to eight control units in the same multidrop (daisy chained) configuration, can be located: 򐂰 Between an ESCON channel and a single control unit. 򐂰 Between an ESCON channel and the first control unit in a multidrop configuration. 򐂰 Between an ESCON Director port and a control unit. A two-port dedicated (static) connection is required within the ESCON Director when attaching to an ESCON conversion (CVC) channel. The distance between the ESCON converter and the control unit cannot exceed the maximum distances supported for the parallel channel I/O environment. Special limitations 32 IBM System z Connectivity Handbook 5444ch03.fm Draft Document for Review July 21, 2010 6:06 pm can exist, depending on the control units or devices used. The maximum channel data rate supported by an ESCON converter is 4.5 MBps, but the actual data rate depends on the attached control unit. The maximum supported distance of the entire ESCON path (with or without an ESCON Director) connecting an ESCON channel to an ESCON converter, using multimode fiber optic cables, is: 򐂰 3 km (1.86 miles) when using 62.5 micron (µm) fiber. 򐂰 2 km (1.24 miles) when using 50 micron (µm) fiber. Note: The ESCON path connecting to an ESCON converter cannot be extended beyond the maximum supported distances using other technologies, such as Wavelength Division Multiplexing (WDM), or ESCON Director XDF feature. The maximum supported distance of the entire ESCON path connecting to an ESCON converter is further restricted by the type of attached control unit. For example, when using 62.5 micron (µm) fiber, an ESCON converter attached to an IBM 3420 model 8 tape drive and IBM 3803 model 2 tape control unit supports: 򐂰 A maximum ESCON link distance of 2.8 km (1.74 miles), without an ESCON Director 򐂰 A maximum entire ESCON path (combined ESCON link) distance of 2.6 km (1.61 miles), with an ESCON Director 3.3 Parallel channel - single path to a control unit Figure 3-1 shows a single path connection to a control unit that is sequentially connected in a daisy chain to other control units. Up to eight control units can be daisy chained on the same parallel channel path. ESCON channel Converter Parallel channel Bus and Tag cables Parallel CU Device Parallel CU Device Parallel CU Device Figure 3-1 Parallel channel to multiple control units 3.4 Parallel channel modes Converters comply to the parallel I/O interface architecture, about which this and the following chapter are a short introduction. The parallel channel interface includes: 򐂰 Byte multiplexer channels 򐂰 Block multiplexer channels Chapter 3. Parallel channel 33 5444ch03.fm Draft Document for Review July 21, 2010 6:06 pm The parallel I/O interface uses two cables called bus and tag. A bus cable carries information (one byte in each direction), and a tag cable indicates the meaning of the information on the bus cable. Each cable itself consists of a bundle of shielded copper coaxial cables with a grounded shield. Bus and tag cables have 48 contacts per connector (signal and ground), which means each control unit connection on the channel interface has 192 contacts. The connection ends with terminator blocks in the last CU. The CU provides the logical capability necessary to operate and control an I/O device, and it adapts the characteristics of each I/O device to the standard form of control provided by the channel. A control unit can be housed separately, or it can be physically and logically integrated with the I/O device. I/O devices are used to provide external storage, to communicate between data processing systems, and to communicate between a data processing system and the external world. The bus cable has a signal line for each bit value. A set of signal lines forming a byte is called Bus Out and carries information from the channel to the control unit (command codes, unit addresses, or data). Another set of signal lines forms another byte, called Bus In, and carries information from the control unit to the channel (status, sense, unit address, or data). The exact meaning of the transmitted bytes is determined by a signal on a tag signal line. The I/O interface addressing facilities can accommodate up to 256 directly addressable I/O devices. Within the given address limitations, and because of timing and electrical considerations, the number of control unit attachment points that can be accommodated on one channel is generally limited to eight. One attachment point can be associated with a single-device control unit, a shared (multi-device) control unit, or multiple independent (integrated) control units. Only one control unit can be connected logically to a parallel I/O interface at any one time. Selection of a control unit by the channel is controlled by a tag signal (Select Out) passing serially through all control units attached to one I/O interface. This serial passing of the selection signal permits each control unit, in turn, to respond to the other signals provided by the channel. Once the control unit is selected, it remains logically connected to the I/O interface until it completes the transfer of the information it has or needs, or until the channel signals the control unit to disconnect. 3.5 Parallel channel elements In this section we discuss parallel channel elements. 3.5.1 Parallel channel There are two main parallel channel modes: byte multiplexer and block multiplexer. Selector channels, which do not disconnect during an I/O operation, are no longer used and have been replaced by block multiplexer channels. 򐂰 Byte multiplexer Byte multiplexer channels are used with slow, unbuffered devices to transfer a small amount of data. For each transfer, the CU must make a request to the channel in order to be reconnected. Typically, these devices are Type-1 and Type-2 CUs. 򐂰 Block multiplexer Block multiplexer channels are used for devices that require a large amount of data at high speed, such as disk and tape devices. This are called Type-3 CUs. Two data transfer protocols are used for block multiplexer channels: Direct-Coupled Interlock (DCI) and data streaming protocol. 34 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch03.fm The data streaming protocol, called a non-interlocked transfer method, provides a data rate of up to 4.5 MBps. 3.5.2 Parallel channel control unit types Parallel channels can connect to three control unit types: 򐂰 Type-1 control unit This type of control unit can control the activity of only a single I/O device at a time. If an I/O operation or chain of I/O operations is in execution, the control unit is unable to handle the initiation of other activity associated with any other attached I/O device. These CUs support I/O devices such as printers, card readers, and teleprocessing units. 򐂰 Type-2 control unit This control unit can control the activity of more than one I/O device at a time without losing control of the ongoing I/O operations. These CUs support I/O devices such as DASD, tapes and communication controllers. 򐂰 Type-3 control unit This control unit can control the activity of more than one I/O device. These CUs support I/O devices such as display device CUs and magnetic tape units. 3.6 References The following publications contain information related to the topics discussed in this chapter. 򐂰 IBM System/360 and System/370 I/O Interface Channel to Control Unit Original Equipment Manufacturer’s Information, GA22-6974 򐂰 z/Architecture Principles of Operation, SA22-7832 Chapter 3. Parallel channel 35 5444ch03.fm 36 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444ch04.fm 4 Chapter 4. Enterprise Systems Connection This chapter describes the Enterprise Systems Connection (ESCON) channel features, and provides information about ESCON I/O connectivity using fiber optic technology. The following topics are covered: 򐂰 ESCON modes and topologies 򐂰 ESCON elements 򐂰 ESCON connectivity © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 37 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm 4.1 Description ESCON provides half-duplex serial bit transmission, in which the communication protocol is implemented through sequences of special control characters and through formatted frames of characters. ESCON utilizes fiber optic cables for data transmission. 4.1.1 ESCON modes and topologies ESCON control units can be connected directly to an ESCON channel (point-to-point), or can be dynamically switched via an ESCON Director (Switched point-to-point or chained ESCON Directors). See Figure 4-1. Point-to-point ESCON link Switched point-to-point ESCON CU ESCON Director ESCON link ESCON CU ESCON links ESCON CU ESCON CU Chained ESCON Directors ESCON Director ESCON link static port connection ESCON Director ESCON link ESCON CU ESCON links ESCON CU ESCON CU Figure 4-1 The ESCON native topologies The switching capabilities allow multiple connections between channels and control units without requiring dedicated physical connections. The point-to-point connections allow physical changes to the I/O configuration concurrently with normal operations on other ESCON paths. An ESCON channel path can include a maximum of two chained ESCON Directors. In a chained configuration, only one of the connections through the ESCON Directors can be a dynamic connection, while the other must be a static connection. The ESCON director providing the dynamic connection can be at either the channel end or the control unit end of the ESCON path. In most configurations, using the dynamic connection in the Director at the control unit end of the ESCON path provides greater configuration flexibility, as shown in Figure 4-1. In order to accommodate parallel channel attached control units and devices, the ESCON conversion mode allows communication from an ESCON channel to parallel 38 IBM System z Connectivity Handbook 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm channel-attached control units operating in block multiplexer or byte multiplexer mode. This is done through an ESCON Converter. See Figure 4-2. The ESCON channel is defined as CVC (block multiplexer mode) or CBY (byte multiplexer mode). ESCON conversion channel ESCON link CHPID type CVC/ CBY ESCON converter Parallel Parallel CU Device Parallel CU Device Parallel CU Device Figure 4-2 The ESCON conversion topology 4.1.2 ESCON elements In general terms, the ESCON I/O interface consists of a set of rules defined by the z/Architecture, physical and logical protocols defined by the ESCON I/O interface architecture, and media specifications that allow the transfer of information in both directions between a z/Architecture channel subsystem and an ESCON attached control unit. The ESCON I/O interface accomplishes the same function as the previous parallel channel interface, and also provides for expanded connectivity, higher performance potential, lower supporting facilities costs, and some additional unique functions. Chapter 4. Enterprise Systems Connection 39 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm ESCON channel An ESCON channel executes commands presented by the standard z/Architecture I/O command set, and it manages its associated link interface (link level/device level) to control bit transmission and reception over the optical medium (physical layer). On a write command, the channel retrieves data from main storage, encodes it, and packages it into device-level frames before sending it on the link. On a read command, the channel performs the opposite operation. See Figure 4-3. Application IOS I/O Requests UCB SSCH I/O Interrupt Channel Subsystem I/O Request (Macros or SVC 0) SSCH (SSID and ORB) CCWs and Data ESCON Channel ESCON Channel ESCON Device Level ESCON Link Level ESCON Physical Layer ESCON I/O Architecture ESCON Topology ESCON CU Figure 4-3 ESCON channel operation flow Note: The IBM zEnterprise 196 is planned to be the last high end server to offer ordering of ESCON channels on new builds, migration offerings, upgrades, and System z exchange programs. Enterprises should begin migrating from ESCON to FICON. Alternate solutions are available for connectivity to ESCON devices. IBM Global Technology Services, through IBM Facilities Cabling Services, offers ESCON to FICON Migration (Offering ID #6948-97D), to help facilitate migration to FICON to simplify and manage a single physical and operational environment while maximizing “green” related savings. For more information see: http://www-935.ibm.com/services/us/index.wss/itservice/igs/a1026000 ESCON link The transmission medium for the ESCON I/O interface is a fiber optic cable. Physically, it consists of a pair of optical fibers that provide two dedicated, unidirectional, serial bit transmission lines. Information in a single optical fiber flows, bit by bit, always in the same direction. The ESCON link data rate is 200 megabits per second (Mbps). At any link interface, one optical fiber of the pair is used to receive data, while the other optical fiber of the pair is used to transmit data. The link is used to attach and interconnect other elements of the ESCON I/O interfaces. 40 IBM System z Connectivity Handbook 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm Note that although there are dedicated fibers for reception and transmission in a single link, full duplex data transfer capabilities are not exploited. The ESCON I/O protocol specifies that for normal I/O operations, frames flow serially in both directions, but in a request/response (half-duplex) fashion. The maximum unrepeated distance of an ESCON link using multimode fiber optic cables is 3 km (1.86 miles) using 62.5 micron fiber. These ESCON link distances can be extended using a variety of solutions. For more information see 11.3.2, “ESCON repeated distance solutions” on page 172. ESCON Director The switching function of ESCON is handled by products called ESCON Directors, which operationally connect channels and control units only for the duration of an I/O operation (dynamic connection). They can switch millions of connections per second, and are the centerpiece of the ESCON topology. Apart from dynamic switching, the ESCON Directors can also be used for static switching of single user control units among different system images (dedicated connection). The switching function of an ESCON Director provides a way of connecting multiple channels and control units without requiring many permanent physical connections. The ESCON Director and its dynamic switching capability is the hub of the ESCON topology. See Figure 4-4. A link connects to an ESCON Director through a port. The number of ports available for connecting external links is implementation-dependent, but architecturally, it cannot exceed 253. The ESCON Director, under frame content control, routes transmissions from one port to any other port in the same ESCON Director, and provides data and command exchange for multiple channels and control units. CPC A Channel Subsystem CPC B CH03 CH03 CH02 CH02 CH01 CH01 C1 C0 C2 CF C3 ESCON Director CU ESCD Port CB CA C7 CU Link Channel Path CC C4 C6 CU CE CD C5 CU Channel Subsystem C8 CU C9 CU CU CU CU Figure 4-4 ESCON Director switching function The ESCON Director (ESCD) temporarily connects ports for transmission. The connection can remain in effect for the duration of the I/O operation while several frames are exchanged Chapter 4. Enterprise Systems Connection 41 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm between the ports. The connection is made (and broken) on request from a frame sent by either the control unit or the channel. ESCD ports are non-blocking, in the sense that there are enough internal paths so that all ports can communicate simultaneously. Signals travel from one port to another, converting from optical signals to electrical and back to optical. This allows for some elasticity in the bit stream (source and destination ports do not have to be locked at exactly the same time), and for repowering the optical signal, balancing for any losses at interfaces or on the fiber cables themselves. The ESCON Director also provides a control unit function that can be used to control port connectivity. Ports can be dedicated to communicate with only one other port, prohibited from communicating with certain other ports, or blocked from communicating altogether. An ESCON channel path may pass through two ESCON Directors. However, the path connection through one Director must be defined on the Director as static, that is, the internal port communication for that link must be dedicated. The same kind of dedicated connection must be used if the port is connected to a conversion channel (CVC and CBY) interface. The control unit function is model-dependent, and can be invoked either by the ESCON Director console or by the control unit port (CUP), which is a reserved port internal to the ESCON Director. The ESCON Manager program product can be used to interface with the control unit port, and provide the user with a high-level, non-implementation-specific, easy to use interface. Important: The IBM ESCON Director 9032 (including all models and features), is withdrawn from marketing. There is no IBM replacement for the IBM 9032. ESCON control units and devices The control unit receives commands from the channel, receives data from and transmits data to the channel, and controls execution of commands and data transfer at associated devices. The control unit attaches to one or multiple links through link interfaces. A control unit can contain multiple images with dedicated or shared physical control unit facilities. The ESCON I/O interface provides addressing for these multiple images. The ESCON I/O interface does not allow multiple control units (or control unit link interfaces) to reside on the same link. However, multiple control units can still be configured to the same channel through an ESCON Director using different switch ports and different links. For ease of migration, many control units can have both ESCON and parallel interfaces, but only one interface type can be connected from the CU to a single system at a time. For example, the paths defined in a DASD CU path group must be either all parallel or all ESCON. ESCON channel-to-channel (CTC) ESCON offers an effective and price-competitive replacement for previous channel-to-channel hardware. With ESCON channels, a user can communicate at channel speed between servers without requiring extra hardware. Chapter 6, “Channel-to-channel” on page 83, covers this connectivity in more detail. ESCON converter An ESCON converter converts S/370 parallel OEMI signals to ESCON protocol; however, it cannot exploit the full functions of ESCON. It is typically used to migrate from the S/370 42 IBM System z Connectivity Handbook 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm parallel OEMI to ESCON, as well as allow connections from existing parallel channel control units to System z servers. Note: The IBM ESCON Converter 9034 is withdrawn from marketing. There is no IBM replacement for the IBM 9034. The PRIZM Protocol Converter Appliance from Optica Technologies Incorporated provides a FICON to ESCON conversion function that has been System z qualified. Qualification letters can be found on the IBM System z I/O Connectivity Web page: http://www-03.ibm.com/systems/z/hardware/connectivity/products/index.html Select the Products tab, then FICON / FCP Connectivity. Scroll down to the "other supported devices" area on the Web page. For more information about the PRIZM Protocol Converter Appliance see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by vendors for products that have not been System z qualified. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. IBM Facilities Cabling Services -- ESCON to FICON migration services can help you take advantage of high-speed FICON to support your upgraded IBM System z environment. And you can keep using your ESCON-attached devices to reduce migration costs. For more information go to: http://www-935.ibm.com/services/us/index.wss/offering/its/c337386u66547p02 4.1.3 Multiple Image Facility The Multiple Image Facility (MIF) enables ESCON channel sharing among logical partitions running on System z servers. Note that ESCON channels cannot be spanned and ESCON conversion CHPIDs (CVC, CBY) cannot be shared. See Chapter 2, “Channel Subsystem overview” on page 13. 4.2 Connectivity The connectivity options for the ESCON I/O interface environment are discussed in this section. The maximum number of ESCON channels supported based on each server can be found in Table 1-2 on page 10. Table 4-1 provides information about System z ESCON channel support, feature codes, and physical specifications listed. Table 4-1 ESCON channel specifications Feature code System z feature name Connector type Cable type Unrepeated max. distance Server 2323 (2324 port activation) 16-port ESCON (1 spare port per card) MT-RJ Connector MM 62.5 µm 3 km (1.24 miles) z196, z10 EC, z10 BC, z9 EC, z9 BC Chapter 4. Enterprise Systems Connection 43 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm For information about distance support for ESCON see, Chapter 11, “Extended distance solutions” on page 167. 4.2.1 ESCON channel support on System z All System z servers support the 16-port ESCON feature (feature code 2323). 16-port ESCON feature The 16-port ESCON feature (feature code 2323) occupies one I/O slot in the System z I/O cages or I/O drawers. Each port on the feature uses a 1300 nm light-emitting diode (LED), designed to be connected to 62.5 micron multimode fiber optic cables. The feature has 16 ports with one CHPID associated with each port, up to a maximum of 15 active ESCON channels per feature. There is a minimum of one spare port per feature to allow for channel sparing in the event of a failure of one of the other ports on the same ESCON channel card. The 16-port ESCON feature allows: 򐂰 򐂰 򐂰 򐂰 Up to 240 channels in the z196, and with RPQ 8P2507 can support up to 360 channels. Up to 1024 channels in the z10 EC, z9 EC, and up to 960 in the z10 EC model E12 Up to 480 channels in the z10 BC Up to 420 channels in the z9 BC (S07) and up to 240 in the z9 BC (R07) The 16-port ESCON feature port utilizes a small form factor optical transceiver that supports a fiber optic connector called MT-RJ. The MT-RJ is an industry standard connector, which has a much smaller profile compared with the original ESCON Duplex connector. The MT-RJ connector, combined with technology consolidation, allows for the much higher density packaging implemented in the 16-port ESCON feature. Note: The 16-port ESCON feature does not support the ESCON Duplex connector. However, ESCON Duplex jumper cables can be reused to connect to the 16-port ESCON feature by installing an MT-RJ/ESCON Duplex conversion kit. This protects the customer’s investment in their existing ESCON Duplex cabling plant. 16-port ESCON adapter port enablement (4-port enablement feature) The 16-port ESCON feature channel cards are initially installed as a pair, and then in increments of one. The 15 active ports on each 16-port ESCON feature are activated in groups of four (ESCON port enablement feature (FC 2324)) via Licensed Internal Code Configuration Control (LIC CC). For z196, z10 and z9 servers, an activated port is assigned a PCHID number by the server Support Element, based on the physical position of the port and feature within the server. See 2.1.5, “Physical Channel ID (PCHID)” on page 17. In most cases, the number of physically installed channels is greater than the number of active channels that are LIC CC-enabled. This is not only because the last ESCON port (J15) of every 16-port ESCON channel card is a spare, but also because several physically installed channels are typically inactive (LIC CC-protected). 44 IBM System z Connectivity Handbook 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm If there is a requirement to increase the number of ESCON channel ports (minimum increment is four), then IBM manufacturing provides an LIC CC diskette to concurrently enable the number of additional ESCON ports. This is shown, using CHPIDs as the example, in Figure 4-5. In this case, no additional hardware is installed. 35 J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 SPARE CHANNEL 57 BF 20 78 35 8A J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 LICCC ENABLEMENT CHPID Number J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 AD D Ch 4 an ES ne C O ls N BF J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 57 20 25 6E SPARE CHANNEL Figure 4-5 4-port ESCON LIC CC enablement on the 16-port ESCON feature An ESCON channel add will never activate the spare channel port. However, if the spare port on a card was previously used because an active port failed, then the add may activate all the remaining ports on that card. If there are not enough inactive ports on existing 16-port ESCON cards installed to satisfy the additional channel order, then IBM manufacturing ships additional 16-port ESCON channel cards and an LIC CC diskette. If there has been multiple sparing on a 16-port ESCON card, and there are not enough good ports on the card to satisfy the customer’s order, the card will be replaced. Chapter 4. Enterprise Systems Connection 45 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm 16-port ESCON channel sparing Failure BF 78 35 8A J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 57 BF 20 25 35 78 6E L NE G AN IN CH PAR S CHPID Number J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 8A J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 SPARE CHANNELS Channel Sparing The last ESCON port on a 16-port ESCON channel card (J15) is initially assigned as a spare port. Should a LIC CC-enabled ESCON port on the card fail, the spare port is used to replace it. This is shown, using CHPIDs as the example, in Figure 4-6. If the initial first spare port (J15) is already in use and a second LIC CC-enabled port fails, then the highest LIC CC-protected port (for instance, J14) is used to spare the failing port. J00 J01 J02 J03 J04 J05 J06 J07 J08 J09 J10 J11 J12 J13 J14 J15 57 20 25 6E SPARE CHANNEL Figure 4-6 16-port ESCON channel sparing Channel port sparing can only occur between ports on the same 16-port ESCON card. That is, a failing ESCON channel port cannot be spared with another port on a different 16-port ESCON card. Channel sparing is a service repair action performed by an IBM Service Representative (IBM SR) using the System z server maintenance package Repair and Verify procedure. If sparing can take place, the IBM SR also moves the external fiber optic cable from the failing port to the spare port. When sparing occurs, the CHPID/PCHID assignment moves to the spare port (CHPID 8A in the example in Figure 4-6). If sparing cannot be performed, the 16-port ESCON card is replaced. In cases where the complete card must be replaced, after a repair the PCHIDs are reset to their initial positions. The IBM SR then uses the Support Element (SE) to obtain the from-to jack assignments for the cables. No IOCDS changes are needed. 4.2.2 Software support Operating systems that support System z also provide support for ESCON, including: 򐂰 z/OS All in-service releases of z/OS; for more information go to: http://www.ibm.com/systems/z/os/zos/support/index.html 򐂰 z/VM All in-service releases of z/VM; for more information go to: http://www.vm.ibm.com/techinfo/lpmigr/vmleos.html 46 IBM System z Connectivity Handbook 5444ch04.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 z/VSE All in-service releases of z/VSE; for more information go to: http://www.ibm.com/servers/eserver/zseries/os/vse/support/support.htm 򐂰 z/Transaction Processing Facility (z/TPF) All in-service releases of z/TPF; for more information go to: http://www.ibm.com/software/htp/tpf/index.html 򐂰 Linux Linux on System z support and downloads; for more information go to: http://www.ibm.com/servers/eserver/zseries/os/linux/dist.html Note: Certain functionality may require specific levels of an operating system or PTFs, or both. 4.3 References The following publications contain information related to the topics discussed in this chapter: 򐂰 ESCON I/O Interface, SA22-7202 򐂰 ESCON Channel-To-Channel Adapter, SA22-7203 򐂰 ESCON Channel-to-Channel Reference, SB10-7034 򐂰 Introducing Enterprise Systems Connection, GA23-0383 򐂰 Enterprise Systems Connection (ESCON) Implementation Guide, SG24-4662 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 Business Class Technical Introduction, SG24-7241 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 Chapter 4. Enterprise Systems Connection 47 5444ch04.fm 48 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444ch05.fm 5 Chapter 5. Fibre Connection This chapter describes the Fibre Connection (FICON) channel offered on System z servers. The term FICON represents the architecture as defined by the International Committee of Information Technology Standards (INCITS), and published as ANSI standards. The following topics are covered: 򐂰 FICON modes and topologies 򐂰 FICON elements 򐂰 FICON connectivity © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 49 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm 5.1 Description FICON provides all the strengths of ESCON while extending the (non-droop) distance from 9 km to 100 km. FICON also increased the link data rate from 17 MBps offered by ESCON to 1, 2, 4, and 8 Gbps with the FICON features. FICON Express8 features auto-negotiate to 2, 4, or 8 Gbps. FICON Express4 features auto-negotiate to 1, 2, or 4 Gbps, and FICON Express2 and FICON Express features auto-negotiate to 1 or 2 Gbps. The FICON implementation enables full-duplex data transfer, so data travels in both directions simultaneously, rather than the half-duplex data transfer of the ESCON implementation. Also, FICON enables multiple concurrent I/O operations, rather than the single-sequential I/O operation of ESCON. Terminology: Throughout this chapter the term FICON is used to refer to the FICON Express, FICON Express2, FICON Express4, and FICON Express8 features, except when the function being described is applicable to a specific FICON feature type. The FICON channel matches data storage and access requirements with the latest technology in servers, control units, and storage devices. FICON channels allow faster and more efficient data transfer, while at the same time allowing you to use their currently installed single mode and multimode fiber optic cabling plant. FICON utilizes long wavelength (LX) and short wavelength (SX) transceivers with multimode and single mode fiber optic media for data transmission. 5.1.1 FICON modes and topologies System z supports the FICON channel to operate in one of three modes: 1. FICON conversion mode (FCV) 2. FICON native mode (FC) 3. Fibre Channel Protocol (FCP) Note: FICON Express2, FICON Express4, and FICON Express8 features do not support FCV mode. FCV mode is available with FICON Express LX feature 2319 (carry forward only on z9 and z10. FICON Express LX is not supported on z196.). The System z10 is the last server family to support FICON Express and FICON Express2. It is intended that the zEnterprise 196 be the last server to support FICON Express4 features, We recommend that you review the usage of your installed FICON Express4 channels and where possible migrate to FICON Express8 channels. 50 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm FICON conversion mode A FICON channel in FICON conversion mode (CHPID type FCV) can access ESCON control units through a FICON Bridge port feature installed in an IBM ESCON Director. See Figure 5-1. FICON Bridge FICON feature FICON Bridge feature FC Link FCV ESCON CU ESCON Director ESCON CU ESCON Links ESCON CU Figure 5-1 FICON conversion mode Important: The IBM ESCON Director (9032-005 including all features) was withdrawn from marketing December 31, 2004. There is no IBM replacement for the 9032-005. The PRIZM Protocol Converter Appliance from Optica Technologies Incorporated provides a FICON to ESCON conversion function that has been System z qualified. Qualification letters can be found on the IBM System z I/O Connectivity Web page: http://www-03.ibm.com/systems/z/hardware/connectivity/products/index.html Select the Products tab, then FICON / FCP Connectivity. Scroll down to the “other supported devices” area on the Web page. For more information about the PRIZM Protocol Converter Appliance, see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by vendors for products that have not been System z qualified. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. IBM Facilities Cabling Services -- ESCON to FICON migration services can help you take advantage of high-speed FICON to support your upgraded IBM System z environment. And you can keep using your ESCON-attached devices to reduce migration costs. For more information go to: http://www-935.ibm.com/services/us/index.wss/offering/its/c337386u66547p02 Chapter 5. Fibre Connection 51 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm FICON native mode As shown in Figure 5-2, a FICON channel in FICON native mode (CHPID type FC) can access FICON native interface control units using the following topologies: 򐂰 Point-to-point (direct connection) 򐂰 Switched point-to-point (via a Fibre Channel switch) 򐂰 Cascaded FICON Directors (through two Fibre Channel switches) Point-to-point FC FICON CU FC Link Switched point-to-point FC FC Link ink FC L FICON CU FICON CU FC Link FC Switch FC Lin k FC Cascaded FICON Directors Site A FC FC Lin k FC Switch Site B ISL ink FC L FC Switch FC Li nk FC L FICON CU FICON CU ink FC Figure 5-2 System z supported FICON native topologies A FICON native channel also supports channel-to-channel communications. The FICON channel at each end of the FICON CTC connection, supporting the FCTC control units, can also communicate with other FICON native control units, such as disk storage devices and tape. Note that at least one end of the FICON CTC connection must be a System z. This is covered in more detail in 6.2.3, “FICON channel-to-channel (FCTC) connectivity” on page 88. Fibre Channel Protocol (FCP) mode A FICON channel in Fibre Channel Protocol mode (CHPID type FCP) can access FCP devices either: 򐂰 Via a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a SCSI device 򐂰 Via a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a Fibre Channel-to-SCSI bridge 52 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm The FICON features provide support of Fibre Channel and Small Computer System Interface (SCSI) devices in z/VM, z/VSE, and Linux on System z. See Figure 5-3. FCP mode FICON-Express2 z9 EC Storage FC switched fabric FCP disks FCP mode FICON Express 8 z10 EC FCP mode FICON Express 4 Storage z10 BC Figure 5-3 System z FCP topologies On a System z server, point-to-point connections can be used to access data stored on devices without the use of a Fibre Channel switch. In addition, an operating system or other standalone program can be IPLed via a point-to-point connection using the SCSI IPL feature. N_Port ID Virtualization (see “N_Port ID Virtualization (NPIV)” on page 56) is not supported with FCP point-to-point. The FCP support allows z/VM, z/VSE, and Linux on System z to access industry-standard SCSI devices. For disk applications, these FCP storage devices utilize Fixed Block (512-byte) sectors rather than Extended Count Key Data (ECKD™) format. 5.1.2 FCP Channel The Fibre Channel Protocol (FC-FCP) standard was developed by the InterNational Committee of Information Technology Standards (INCITS), and published as ANSI standards. The System z FCP I/O architecture conforms to the FC standards specified by the INCITS. More detailed information about the FC standards can be obtained from the following Web sites: http://www.t10.org http://www.t11.org FICON channels in FCP mode provide full fabric attachment of SCSI devices to the operating system images, using the Fibre Channel Protocol, and provide point-to-point attachment of SCSI devices. This allows z/VM, z/VSE, and Linux on System z to access industry-standard SCSI storage controllers and devices. Chapter 5. Fibre Connection 53 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm The FCP channel full fabric support means that multiple numbers of Directors/switches can be placed between the System z server and SCSI device, thereby allowing many “hops” through a storage area network (SAN) and providing improved utilization of intersite connected resources and infrastructure. This expanded attachability is intended to provide more choices for storage solutions, or the ability to use existing storage devices. This can facilitate the consolidation of UNIX® server farms onto System z servers, protecting investments in SCSI-based storage. For a list of switches, storage controllers, and devices that have been verified to work in a Fibre Channel network attached to FCP channel, and for specific software requirements to support FCP and SCSI controllers or devices, see: http://www.ibm.com/systems/z/connectivity/products/fc.html FICON channels in FCP mode are based on the Fibre Channel standards defined by the INCITS, and published as ANSI standards. FCP is an upper layer Fibre Channel mapping of SCSI on a common stack of Fibre Channel physical and logical communication layers. SCSI is supported by a wide range of controllers and devices, which complement the classical storage attachment capability through FICON and ESCON channels. FCP is the base for industry-standard Fibre Channel networks or Storage Area Networks (SANs). Fibre Channel networks consist of servers, storage controllers, and devices as end nodes, interconnected by Fibre Channel switches, Directors, and hubs. Switches and Directors are used to build Fibre Channel networks or fabrics. Fibre Channel loops (FC-AL) can be constructed using Fibre Channel hubs. In addition, different types of bridges and routers can be used to connect devices with different interfaces (like parallel SCSI). All of these interconnects can be combined in the same network. SCSI has been implemented by many vendors in a large number of different types of storage controllers and devices. These controllers and devices have been widely accepted in the marketplace, and have proved able to meet the reliability, availability, and serviceability (RAS) requirements in many environments. FICON channels in FCP mode use the Queued Direct Input/Output (QDIO) architecture for communication with the operating system. The QDIO architecture for FCP channels derives from the QDIO architecture that initially was defined for the OSA-Express features and for HiperSockets communications. FCP channels do not use control devices; instead data devices that represent QDIO queue pairs are defined, consisting of a request queue and a response queue. Each queue pair represents a communication path between an operating system and the FCP channel. It allows an operating system to send FCP requests to the FCP channel via the request queue. The FCP channel uses the response queue to pass completion indications and unsolicited status indications to the operating system. See Figure 5-4 on page 55 for a graphical representation of this. HCD/IOCP is used to define the FCP channel type and QDIO data devices; however, there is no definition requirement for the Fibre Channel storage controllers and devices, nor the Fibre Channel interconnect units like switches, Directors, and bridges. The FCP industry-standard architecture requires that the Fibre Channel devices (end nodes) in a Fibre Channel network are addressed using World Wide Names (WWNs), Fibre Channel Identifiers (IDs), and Logical Unit Numbers (LUNs). These addresses are configured on an operating system level, and passed to the FCP channel together with the corresponding Fibre Channel I/O or service request via a logical QDIO device (queue). 54 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm Figure 5-4 summarizes the necessary FCP I/O definitions and compares them to FICON and ESCON I/O definitions. Classical System z I/O Definitions Device number Host Program IOCP CHPID Switch Link CUADD FICON/ESCON CHANNEL System z FCP (SCSI) I/O Definitions Device number WWPN LUN Control Unit FCP CHANNEL FCP SAN CHPID IODEVICE Address Device number defintions identify the communication path to the FCP channel and QDIO device queue WWPN (8 Byte) World Wide Port Name is used to query the controller port connection in the fabric LUN (8 Byte) Logical Unit Number identifies the device Device number definitions identify the communication path to the device Device IOCP QDIO device queues Director/ FC Switch Director/ FC Switch Host Program Controller Device Figure 5-4 I/O definition comparison (FCP to FICON and ESCON) Channel and device sharing An FCP channel can be shared between multiple Linux operating systems, each running in a logical partition or as a guest operating system under z/VM. To access the FCP channel, each operating system needs its own QDIO queue pair, defined as a data device on an FCP channel in the HCD/IOCP. Each FCP channel can support up to 480 QDIO queue pairs with the z196, z10 and z9 servers. This allows each FCP channel to be shared among 480 operating system instances (maximum of 252 guests per logical partition). Host operating systems sharing access to an FCP channel can establish, in total, up to 2048 concurrent connections to up to 512 different remote Fibre Channel ports associated with Fibre Channel controllers. The total number of concurrent connections to end devices, identified by logical unit numbers (LUNs), must not exceed 4096. While multiple operating systems can concurrently access the same remote Fibre Channel port via a single FCP channel, Fibre Channel devices (identified by their LUNs) can only be serially re-used. In order for two or more unique operating system instances to share concurrent access to a single Fibre Channel or SCSI device (LUN), each of these operating systems will have to access this device through a different FCP channel. Should two or more unique operating system instances attempt to share concurrent access to a single Fibre Channel or SCSI device (LUN) over the same FCP channel, a LUN sharing conflict will occur and errors will result. A way to alleviate this sharing conflict on System z servers is to use N_Port ID Virtualization. Chapter 5. Fibre Connection 55 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm N_Port ID Virtualization (NPIV) NPIV is designed to allow the sharing of a single physical FCP channel among operating system images, whether in logical partitions or as z/VM guests in virtual machines. This is achieved by assigning a unique World Wide Port Name (WWPN) to each operating system connected to the FCP channel. In turn, each operating system appears to have its own distinct WWPN in the SAN environment, thus enabling separation of the associated FCP traffic on the channel. Access controls based on the assigned WWPN can be applied in the SAN environment using standard mechanisms such as zoning in FC switches and Logical Unit Number (LUN) masking in the storage controllers. NPIV is exclusive to the System z servers and is the preferred method for sharing SCSI devices among multiple operating system images when using a single physical FCP channel. Note: NPIV exploitation requires an FC switch that supports the NPIV standard. It is not supported in a point-to-point topology. The WWPNs assigned to an FCP channel can be displayed using the Display NPIV Configuration task on the HMC/SE of the System z servers. For details, refer to Introducing N_Port Identifier Virtualization for IBM System z9, REDP-4125. Worldwide port name (WWPN) prediction tool The worldwide port name prediction tool assigns WWPNs to each Fibre Channel Protocol (FCP) channel/port using the same WWPN assignment algorithms that a system uses when assigning WWPNs for channels utilizing NPIV. Thus, the SAN can be set up in advance, allowing operations to proceed much faster once the server is installed. The WWPN prediction tool can calculate and show WWPNs for both virtual and physical ports ahead of system installation. This means, the SAN configuration can be retained instead of altered by assigning a WWPN to physical FCP ports when a FICON feature is replaced. The WWPN prediction tool takes a .csv file containing the FCP-specific I/O device definitions and creates the WWPN assignments that are required to set up the SAN. A binary configuration file that can be imported later by the system is also created. The .csv file can either be created manually or exported from the Hardware Configuration Definition/Hardware Configuration Manager (HCD/HCM). FCP SCSI IPL feature enabler (FC 9904) This function allows you to IPL an operating system from an FCP channel attached disk, to execute either in a logical partition or as a guest operating system under z/VM. In particular, SCSI IPL is able to directly IPL a Linux operating system that has previously been installed on a SCSI disk. Thus, there is no need for a classical channel (ESCON or FICON) attached device, such as an ECKD disk control unit, in order to install and IPL a Linux operating system. The IPL device is identified by its Storage Area Network (SAN) address, consisting of the worldwide port number (WWPN) of the disk controller and the Logical Unit Number (LUN) of the IPL device. SCSI IPL is supported in conjunction with: 򐂰 FCP access control 򐂰 Point-to-point connections 򐂰 N-Port ID Virtualization (NPIV) 56 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch05.fm You can also IPL a standalone-dump program from an FCP channel-attached SCSI disk. The standalone-dump program can also store the generated dump data on SCSI disk. To use this function with an FCP channel, a no-charge orderable platform feature (FC 9904) is required. z/VM support of SCSI IPL allows Linux and other guest operating systems that support SCSI IPL to be IPLed from FCP-attached SCSI disk, when z/VM is running on a System z server equipped with the SCSI IPL function. Therefore, Linux guests can be started and run completely from FCP channel-attached disk in your hardware configuration. Note: This function is standard on the zEnterprise 196 and System z10, no need to order FC 9904. For additional information about FCP SCSI IPL refer to Linux on System z: Fibre Channel Protocol Implementation Guide, SG24-6344. FCP multipathing concept With the channel subsystem on System z servers, multipath access to disk subsystems is a basic feature. Both ESCON and FICON connections support multiple hardware paths to any physical disk device. The System z handles multipathing invisibly by the operating system. In the ESCON and FICON channels, a single device is presented to the operating system to do I/O operations on. Multipathing happens under Channel Subsystem control. Multipathing over FCP is different. With FCP multipathing on Linux on System z, each path to each LUN appears to the operating system as a separate device. For example, if there are four paths to five LUNs, the Linux system sees 20 SCSI devices. This means that there must be another layer of code between the Linux filesystem layer and the SCSI subsystem. This extra layer handles all of the coordination between the raw paths and the higher level file subsystem. Its implementation is Linux distribution dependent. On Red Hat Linux this layer is handled by RAID (Redundant Array of Independent Disks) software and mdadm (multiple device administration). On SUSE this layer is handled by LVM (Logical Volume Manager). IFCP multipathing. In order to get the most benefit from multipathing, multiple FCP CHPIDs and one storage system host adapter per CHPID should be used. FCP multipathing is managed at the Linux system level. There is no global multipathing scheduler that works across the entire System z for FCP. FCP access control The ability to control access to nodes and devices is provided as a function in switches and controllers, and is called LUN masking and zoning. LUN masking and zoning can be used to prevent servers from accessing storage they are not permitted to access. 򐂰 LUN masking A LUN represents a portion of a controller, such as a disk device. With the use of LUNs, a controller can be logically divided into independent elements or groups of elements. Access to these LUNs can be restricted to distinctive WWPNs as part of the controller configuration. This method is known as LUN masking. 򐂰 Zoning Segmentation of a switched fabric is achieved through zoning. It can be used to fence off certain portions of the switched fabric, allowing only the members of a zone to communicate within that zone. All others attempting to access from outside that zone are rejected. Chapter 5. Fibre Connection 57 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm I/O devices The System z FCP channel implements the FCP standard as defined by the INCITS Fibre Channel Protocol for SCSI (FC-FCP), and Fibre Channel Protocol for SCSI, Second Version (FCP-2), as well as the relevant protocols for the SCSI-2 and SCSI-3 protocol suites. Theoretically, each device conforming to these protocols should work when attached to a System z FCP channel. However, experience tells us that there are small deviations in the implementations of these protocols. Also, for certain types of FCP and SCSI controllers and devices, specific drivers in the operating system may be required in order to exploit all the capabilities of the controller or device, or to cope with unique characteristics or deficiencies of the device. Note: We recommend doing appropriate conformance and interoperability testing to verify that a particular storage controller or device can be attached to a System z FCP channel in a particular configuration (that is, attached via a particular type of Fibre Channel switch, Director, or via point-to-point connection). Hardware assists A complementary virtualization technology is available for z196, z10 and z9 servers, which includes: 򐂰 QDIO Buffer-State Management (QEBSM): Two hardware instructions designed to help eliminate the overhead of hypervisor interception 򐂰 Host Page-Management Assist (HPMA): An interface to the z/VM main storage management function designed to allow the hardware to assign, lock, and unlock page frames without z/VM hypervisor assistance These hardware assists allow a cooperating guest operating system to initiate QDIO operations directly to the applicable channel, without interception by z/VM, thereby helping to provide additional performance improvements. This support is integrated in the z196, z10 and z9 servers; however, you should refer to the appropriate 2817DEVICE, 2098DEVICE, 2097DEVICE, 2096DEVICE, and 2094DEVICE, Preventative Service Planning (PSP) buckets prior to implementation. 5.1.3 FCP and FICON mode characteristics Probably the single largest difference between the FICON channel (FC and FCV) and FCP channel mode types is the treatment of data access control and security. FICON (and ESCON) channels rely on Multiple Image Facility (MIF) to address concerns regarding shared channels and devices. MIF provides ultra-high access control and security of data so that one operating system image and its data requests cannot interfere with another operating system’s data requests. With the introduction of the System z servers, MIF continues this ultra-high level of access control and security across Channel Subsystems (CSSs). Linux guest operating systems under z/VM can have access to an FCP channel defined to the z/VM operating system. An FCP channel can also be shared, through MIF, between Linux Logical Partitions and z/VM Logical Partitions with Linux guests. The FCP industry-standard architecture does not exploit the data access control and security functions of MIF. As a result, FCP has the following limitations: 򐂰 Channel sharing When N_Port ID Virtualization (NPIV) is not implemented, multiple Linux images share an FCP channel and all of the Linux images have connectivity to all of the devices connected to the FCP fabric. Since the Linux images all share the same FCP channel, all of the Linux 58 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch05.fm images use the same worldwide port name to enter the fabric and thus are indistinguishable from each other within the fabric. Hence, the use of zoning in switches and LUN-masking in controllers cannot be effective in creating appropriate access controls among the Linux images. By using NPIV, each operating system sharing an FCP channel is assigned a unique WWPN. The WWPN can be used for device-level access control in storage controllers (LUN masking), as well as switch-level access control (zoning). 򐂰 Device sharing Without using NPIV, an FCP channel prevents logical units from being opened by more than one Linux image at a time. Access is on a first-come, first-served basis. This is done to prevent problems with concurrent access from Linux images that are sharing the same FCP channel (and thus the same worldwide port name). This usage-serialization means that one Linux image can block other Linux images from accessing the data on one or more logical units, unless the sharing images (z/VM guests) are all “well-behaved” and not in contention. FICON and FCP have some other significant differences. Some of these are fundamental to System z, others are fundamental to the two channel architectures, and still others may be dependent on the operating system and the device being attached. Without taking the operating system and the storage device into consideration, I/O connectivity via System z FCP and FICON channels has the following differences: 򐂰 Direct connection With all FICON features on z196, z10 and z9 servers, storage controllers can be directly connected to the channel via point-to-point attachment, when in FCP mode. There is no need for a director or switch between the FCP channel and storage controllers. While N_Port ID Virtualization is supported in switch topology, it is not supported in a point-to-point topology. 򐂰 Switch topology FCP channels will support full fabric connectivity, meaning that a number of Directors/switches can be used between a System z server and the device. With the FICON cascaded Directors support, the FICON storage network topology is limited to a two-Director, single-hop configuration. Enterprise fabric: The use of cascaded FICON Directors would ensure the implementation of a high-integrity fabric. For FCP, a high-integrity fabric solution is not mandatory – although we strongly recommended it. For example, should an FCP Inter-Switch Link (ISL) be moved, data could potentially be sent to the wrong path without notification. This type of error would not happen on an enterprise fabric with FICON. 򐂰 Transmission data checking When a transmission is sent via an FCP channel, due to its full-fabric capability, FCP performs intermediate data checking for each leg of that transmission. FICON (and ESCON) also perform intermediate data checking. 򐂰 Serviceability – Licensed Internal Code (LIC) updates and z196, z10 and z9 FCP concurrent patch FICON channels, when configured as CHPID type FCP, support concurrent patches, allowing the application of a LIC without requiring a configuration of off/on. This is an exclusive FCP availability feature, available with all FICON features. – The FICON Express8 and FICON Express4 features have Small Form Factor Pluggable (SFP) optics to permit each channel to be individually serviced in the event of a fiber optic module failure. The traffic on the other channels on the same feature can continue to flow if a channel requires servicing. Chapter 5. Fibre Connection 59 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 Problem determination – Request Node Identification (RNID) RNID is designed to assist with the isolation of FICON cabling-detected errors. In a fiber optic environment, with extended distances, resolution of fiber optic cabling problems can be a challenge. To help facilitate resolution, the operating system can now request the RNID data for each device or control unit attached to native FICON channels and display the RNID data using an operator command. RNID is exclusive to the zEnterprise 196, System z10 and System z9, and is supported by all FICON features (CHPID type FC), and by z/OS and z/OS.e. – Link incident reporting Link incident reporting is integral to the FICON (and ESCON) architecture. When a problem on a link occurs, this mechanism identifies the two connected nodes between which the problem occurred, potentially leading to faster problem determination and service. For FCP, link incident reporting is not a requirement for the architecture (though it may be offered as an optional switch function). Consequently, important problem determination information may not be available should a problem occur on an FCP link. System z allows z/OS to register for all FICON link incident records. This improves the ability to capture data for link incident analysis across multiple servers. This function is supported with z/OS version 1 release 7 or higher. – Simplified problem determination To more quickly detect fiber optic cabling problems in a Storage Area Network all FICON channel error information is forwarded to the HMC. Thus facilitating detection and reporting trends and thresholds for the channels with aggregate views including data from multiple systems. Problem determination can now be simplified by using the HMC to more quickly pinpoint fiber optic cabling issues in your Storage Area Network (SAN) fabric without IBM service personnel involvement. All FICON channel error information is forwarded to the HMC where it is analyzed to help detect and report the trends and thresholds for all FICON channels on System z10. The Fibre Channel Analyzer task on the HMC can be used to display analyzed information about errors on FICON channels (CHPID type FC) of attached Support Elements. Data includes information about the PCHID, CHPID, channel type, source link address, and destination link address of where the error occurred. This report shows an aggregate view of the data and can span multiple systems. – FICON purge path extended The purge path extended function provides enhanced capability for FICON problem determination. FICON purge path error-recovery function is extended so that it transfers error-related data and statistics between the channel and entry switch and the control unit and its entry switch to the host operating system. FICON purge path extended exploitation requires a switch or device that supports this function. 򐂰 FICON error recovery System z servers and z/OS and z/OS.e I/O recovery processing is designed to allow for the system to detect switch/director fabric problems that may cause FICON links to fail and recover multiple times in a short period of time. This allows the system to detect these conditions and keep an affected path offline until an operator action is taken. This is expected to help limit the performance impacts of switch/director fabric problems. The FICON error recovery function is available in z/OS. 60 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm FICON performance The latest FICON and FCP performance information can be found at the IBM System z I/O Connectivity Web page: http://www-03.ibm.com/systems/z/hardware/connectivity/products/index.html 5.2 FICON elements FICON enables multiple concurrent I/O operations to occur simultaneously to multiple control units. This is one of the fundamental differences between FICON and ESCON, with its sequential I/O operations. FICON channels also permit intermixing of large and small I/O operations on the same link, a significant improvement compared with ESCON channels. The data center I/O configuration now has increased connectivity flexibility because of the increased I/O rate, increased bandwidth, and multiplexing of mixed workloads. FICON channel FICON channel architecture is compatible with: 򐂰 Fibre Channel Physical and Signaling standard (FC-FS) 򐂰 Fibre Channel Switch Fabric and Switch Control Requirements (FC-SW) 򐂰 Fibre Channel Single-Byte-3 (FC-SB-3) and Fibre Channel Single-Byte-4 (FC-SB-4) standards When an application performs an I/O operation to a device represented by a Unit Control Block (UCB), it initiates an I/O request using macros or Supervisor Call to the Input Output Supervisor (IOS). The application or access method also provides the channel program (Channel Command Words - CCWs), and additional parameters in the Operation Request Block (ORB). This request is queued on the UCB. The IOS will service the request from the UCB on a priority basis. The IOS then issues a Start Subchannel (SSCH) instruction, with the Subsystem Identification word (SSID) representing the device and the ORB as operands. The Channel Subsystem (CSS) is signaled to perform the operation. See Figure 5-5. I/O Requests Application IOS UCB SSCH I/O Interrupt Channel Subsystem I/O Request (Macros or SVC 0) SSCH (SSID and ORB) CCWs and Data Fibre Channel Channel FC-4 (FC-SB) FC-SB-3 or FC-SB-4 protocol (FICON-IUs) FC-3 (Services) FC-2 (Framing) FC-1 (Encode/ Decode) FC-2 (FC-PH and FC-FS) frames FC-0 (Optics) Fibre Channel Fabric FC-SB-3 or FC-SB-4(FICON) protocol in FC-FS frames FICON CU Figure 5-5 FICON channel operation flow Chapter 5. Fibre Connection 61 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm The most appropriate FICON channel is selected by the CSS. The FICON channel will fetch channel programs (CCWs) prepared by the application, fetch data from memory (write), or store data into memory (read) and present status of operation to application (I/O interrupt). The z/Architecture channel commands, the data, and status are packaged by the FICON channel into FC-SB-3 or FC-SB-4 (FC-4 layer) Information Units (IUs). IUs from several different I/O operations to the same or different control units and devices are multiplexed or demultiplexed by the FC-2 layer (framing). These FC-2 frames (with encapsulated FC-SB-3 or FC-SB-4 IUs) are encoded or decoded by the FC-1 layer (Encode/Decode) and sent or received to or from the FC-0 Fiber optic medium (optics). A fundamental difference with ESCON is the CCW chaining capability of the FICON architecture. While ESCON channel program operation requires a Channel End/Device End (CE/DE) after execution of each CCW, FICON supports CCW chaining without requiring a CE/DE at the completion of each CCW operation. The ESCON channel transfers the CCW to the control unit and waits for a CE/DE presented by the control unit after execution of the CCW by the device (CCW interlock). After receiving CE/DE for the previous CCW, the next CCW is transferred to the control unit for execution. On a FICON channel, CCWs are transferred to the control unit without waiting for the first command response (CMR) from the control unit or for a CE/DE after each CCW execution. The device presents a logical End to the control unit after each CCW execution. If the last CCW of the CCW chain has been executed by the device, the control unit presents CE/DE to the channel. Figure 5-6 shows a CCW operation on a FICON channel that exploits CCW chaining. FICON Channel Control Unit CCW1 CCW2 CCW3 CCW1 CCW2 CCW3 CCW4 CCW5 CCW6 ----- CCW4 CCW5 CCW6 ----CE/DE CMR Status Except Figure 5-6 CCW chaining High performance FICON for System z (zHPF) zHPF is an enhancement of the FICON channel architecture and is compatible with: 򐂰 Fibre Channel Physical and Signaling standard (FC-FS) 򐂰 Fibre Channel Switch Fabric and Switch Control Requirements (FC-SW) 򐂰 Fibre Channel Single-Byte-4 (FC-SB-4) standards Exploiting zHPF by the FICON channel, the z/OS operating system, and the control unit will reduce the FICON channel overhead. This is achieved by protocol optimization and reducing 62 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm the number of Information Units (IUs) processed, resulting in more efficient usage of the fiber link. The FICON Express8, FICON Express4, and FICON Express2 features support both the existing FICON architecture and the zHPF architecture. From the z/OS point of view the existing FICON architecture is called command mode, and zHPF architecture is called transport mode. A parameter in the Operation Request Block (ORB) is used to determine whether the FICON channel is running in command or transport mode. The mode used for an I/O operation depends on the control unit supporting zHPF and settings in the z/OS operating system. An IECIOSxx parameter and SETIOS commands in z/OS can enable or disable zHPF. Support is also added for the D IOS,ZHPF system command to indicate whether zHPF is enabled, disabled, or not supported on the server. During link initialization both the channel and the control unit indicate whether they support zHPF. In the response to the Request Node ID (RNID) Extended Link Services (ELS) the Process Login (PRLI) support indicator is presented. If PRLI is supported, the channel sends a PRLI ELS. The PRLI response then indicates that zHPF is supported by the CU. Similar to the existing FICON channel architecture described on page 61, the application or access method provides the channel program (CCWs) and parameters in the Operation Request Block (ORB). Bit 13 in word 1 of the ORB specifies how to handle the channel program in either command mode or transport mode. The way zHPF (transport mode) manages CCW operation is significantly different from the CCW operation for the existing FICON architecture (command mode) shown in Figure 5-7. While in command mode each single CCW is sent to the control unit for execution, in transport mode all CCWs are sent over the link in one single frame to the control unit. Certain complex CCW chains are not supported by zHPF. Figure 5-7 shows an example of the optimization by a zHPF (transport mode) read operation. FICON Channel Control Unit CCW1 (Read 4 KB) + CCW2 (Read 4 KB) + CCW3 (Read 4 KB) + CCW4 (Read 4 KB) Data (16 KB) Status (CE/DE) Figure 5-7 High performance FICON read operation The channel sends all the required CCWs and read operations of 4 KB of data in one single frame to the control unit. The control unit transfers the requested data over the link to the channel, followed by a CE/DE if the operation was successful. Less overhead is generated compared to the existing FICON architecture. Chapter 5. Fibre Connection 63 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm The same reduction of frames and open exchanges for a zHPF (transport mode) write operation are shown in Figure 5-8. FICON Channel Control Unit CCW1 (Write 4 KB) + CCW2 (Write 4 KB) + CCW3 (Write 4 KB) + CCW4 (Write 4 KB) XFER Ready (16 KB) Data (16 KB) Status (CE/DE) Figure 5-8 High performance FICON write operation The channel sends all the required CCWs and write operations of 4 KB of data in one frame to the control unit. The control unit responds with a XFER ready when it is ready to receive the data. The channel then sends the 16 KB of data to the control unit. If the control unit successfully received the data and finished the write operation, then the CE/DE status will be sent by the control unit to indicate the completion of the write operation. zEnterprise 196 and System z10 severs allow up to 256 tracks of data to be transferred in one single operation. This allows the channel to fully exploit the bandwidth of FICON channels, resulting in higher throughputs and lower response times. It applies exclusively to the FICON Express8 and FICON Express4 features when configured as CHPID type FC. The FICON control unit must support zHPF. For a more detailed description of the FICON channel protocol and zHPF refer to FICON Planning and Implementation Guide, SG24-6497. Platform and name server registration in FICON channel All FICON features on the z196 and z10 servers support platform and name server registration to the fabric if the FICON feature is defined as CHPID type FC. Information about the channels connected to a fabric, if registered, will allow other nodes or SAN managers to query the name server to determine what is connected to the fabric. The attributes that are registered for the z196 and z10 servers are as follows: 򐂰 Platform information – World Wide Node name (WWN): This is the node name of the platform and is the same for all channels belonging to the platform. – Platform type. – Host computer type. – Platform name: The platform name includes Vendor ID, product ID, and vendor specific data from the node descriptor. 򐂰 Channel information 64 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 򐂰 򐂰 򐂰 򐂰 5444ch05.fm World Wide Port Name (WWPN) Port type (N_Port_ID) FC-4 types supported Classes of service supported by the channel The platform and name server registration service is defined in the Fibre Channel - Generic Services 4 (FC-GS-4) standard. Open exchanges An open exchange is part of FICON (and FC) terminology. An open exchange represents an I/O operation in progress over the channel. Many I/O operations can be in progress over FICON channels at any one time. For example, a disk I/O operation might temporarily disconnect from the channel while performing a seek operation or while waiting for a disk rotation. During this disconnect time, other I/O operations can be managed, as follows: 򐂰 Command mode open exchanges In command mode, the number of open exchanges is limited by the FICON Express feature. FICON Express8, FICON Express4, and FICON Express2 allow up to 64 open exchanges. One open exchange (actually, this is an exchange pair) in command mode is the same as one I/O operation in progress. 򐂰 Transport mode open exchanges In transport mode, one exchange is sent from the channel to the control unit. Then the same exchange ID is sent back from the control unit to the channel to complete the I/O operation. The maximum number of simultaneous exchanges that the channel can have open with the CU is up to 750 exchanges. The CU sets the maximum number of exchanges in the status area of the transport mode response IU (the default number is 64 and can be increased or decreased). In addition, FICON channels can multiplex data transfer for several devices at the same time. This also allows workloads with low to moderate control unit cache hit ratios to achieve higher levels of activity rates per channel. If the open exchange limit is reached, additional I/O operations are refused by the channel. This can result in queuing and retries by the operating system. Extended distances Degradation of performance at extended distances can be avoided by implementing an enhancement to the industry standard FICON architecture (FC-SB-3). The enhancement is a protocol for persistent information unit (IU) pacing. Control units that exploit the architecture can increase the pace count, meaning the number of information units allowed to be underway between the channel and the control unit. Extended distance FICON channels remember the last pacing information and use this information for subsequent operations, thus avoiding performance degradation at the start of a new operation. Information unit pacing helps to optimize the link utilization and simplifies the requirements for channel extension equipment because more commands can be in flight. Extended distance FICON is transparent to the operating systems and is applicable to all FICON features defined with CHPID type FC. Support is provided on the IBM System Storage™ DS8000® with the appropriate LMC level and is exclusive to zEnterprise 196 and System z10 servers. Chapter 5. Fibre Connection 65 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm Modified Indirect Data Address Word (MIDAW) On System z, the Modified Indirect Data Address Word provides alternatives to using CCW data chaining in channel programs. The MIDAW facility has been added to z/Architecture and can coexist with the current CCW IDAW facility. MIDAW is designed to decrease channel, fabric, and control unit overhead by reducing the number of CCWs and frames processed, and allows scattering of data in memory for noncontiguous real pages. While the CCW IDAW function requires all but the first and last IDAW in a list to deal with complete 2K or 4K units of data, the MIDAW facility allows page boundary crossing on either 2K or 4K boundaries. This allows access to data buffers anywhere in a 64-bit buffer space. Figure 5-9 is an example of MIDAW usage. CCW with IDAW flag set (with ORB specifying 4K blocks) CCWs cmd 04 IDAW address Any IDAWs after first one must begin and end on a 4K boundary Real address Real address IDAWs IDAWs can start at any address within 4K block Real address IDAWs usage limited because of addressing requirements The last IDAW can end at any address in a 4K block CCW with MIDAW flag set CCW MIDAWs cmd 01 MIDAW address reserved 2k Real address reserved 32 Real address 1k Real address reserved L MIDAWs removes the addressing restrictions of IDAWs enabling scattered reads/writes with one CCW IDAWs can not be used for scattered reads / writes MIDAWs used for extended format data sets can potentially reduce number of CCWs, eliminating some CCW processing improving channel performance Figure 5-9 IDAW and MIDAW usage The use of MIDAWs is indicated by the MIDAW bit (flag bit 7) in the CCW. The last MIDAW in the list will have a last flag set, indicated by L in Figure 5-9. MIDAW provides significant performance benefits, especially when processing extended format data sets with FICON channels. FICON link The transmission medium for the FICON interface is a fiber optic cable. Physically, it is a pair of optical fibers that provide two dedicated, unidirectional, serial-bit transmission lines. Information in a single optical fiber flows, bit by bit, in one direction. At any link interface, one optical fiber is used to receive data while the other is used to transmit data. Full duplex capabilities are exploited for data transfer. The Fibre Channel Standard (FCS) protocol specifies that for normal I/O operations frames flow serially in both directions, allowing several concurrent read and write I/O operations on the same link. The link data rate is: 򐂰 1 or 2 Gbps for FICON Express and FICON Express2 channels 66 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 1, 2, or 4 Gbps for FICON Express4 channels 򐂰 2, 4, or 8 Gbps for FICON Express8 channels Whether these link data rates can be achieved depends on many factors, like transfer sizes and access methods used, to name a few. The 2 Gbps and 4 Gbps link capability is auto-negotiated between the server and FICON Director, as well as the Director and CUs, and is transparent to the operating system and application. In general, the FICON channels and the FICON Director or CU communicate and agree upon either a 1 Gbps, 2 Gbps, 4 Gbps, or 8 Gbps (100 MBps, 200 MBps, 400 MBps, or 800 MBps) link speed. This speed determination is based on the supported speeds in the server feature, FICON Director, and CU. Note: The link speed is the theoretical maximum unidirectional bandwidth capability of the fiber optic link. The actual data rate of the link (whether it is measured in I/O operations per second, or MBps) will depend on the type of workload, fiber infrastructure, and storage devices in place. FICON LX features utilize long wavelength (LX) transceivers and 9 micron single mode fiber optic media (cables and trunks) for data transmission. FICON SX features utilize short wavelength (SX) transceivers and 50 or 62.5 micron multimode fiber optic media (cables and trunks) for data transmission. A transceiver is a transmitter and receiver. The transmitter converts electrical signals to optical signals to be sent over the fiber optic media. The receiver converts optical signals to electrical signals to be sent through the server, Director, or CU. FICON Bridge The FICON Bridge topology is intended to help provide investment protection for currently installed ESCON control units. The IBM 9032 Model 005 ESCON Director is the only Director that supports long wavelength FICON links through the use of a FICON Bridge (one port) feature. Important: The IBM ESCON Director 9032 Model 5 (including all models and features), is withdrawn from marketing. There is no IBM replacement for the IBM 9032. The PRIZM Protocol Converter Appliance from Optica Technologies Incorporated provides a FICON to ESCON conversion function that has been System z qualified. Qualification letters can be found on the IBM System z I/O Connectivity Web page: http://www-03.ibm.com/systems/z/hardware/connectivity/products/index.html Select the Products tab, then FICON / FCP Connectivity. Scroll down to the “other supported devices” area on the Web page. For more information about the PRIZM Protocol Converter Appliance, see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by vendors for products that have not been System z qualified. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. IBM Facilities Cabling Services -- ESCON to FICON migration services can help you take advantage of high-speed FICON to support your upgraded IBM System z environment. And you can keep using your ESCON-attached devices to reduce migration costs. For more information go to: Chapter 5. Fibre Connection 67 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm http://www-935.ibm.com/services/us/index.wss/offering/its/c337386u66547p02 FICON/Fibre Channel switch The Fibre Channel Switch (FC-SW) supports packet-switching, which provides better utilization than the circuit-switching in the ESCON Director. It allows up to 321 simultaneous concurrent I/O operations (read and write) from multiple FICON-capable systems to multiple FICON control units. Figure 5-10 shows a conceptual view of frame processing in a switched point-to-point configuration for multi-system and multi-control unit environments. FICON Channel (FC) FIC ON Fra me s FIC ON Fra me s FC switch s me Fra N O FIC es ram NF O FIC FC links FC links O FIC FICON Channel (FC) FIC O es ram NF ON FIC s me Fra F_Port FICON CU & I/O FIC O NF ram es NF ram es FICON Adapter FICON CU & I/O Figure 5-10 FICON switch function FICON support of cascaded Directors FICON native channels on System z servers support cascaded Fibre Channel Directors. This support is for a two-Director configuration only. With cascading, a FICON native channel, or a FICON native channel with channel-to-channel (FCTC) function, can connect a server to a device or other server via two native connected Fibre Channel Directors. This cascaded Director support is for all native FICON channels implemented on System z servers. FICON support of cascaded Directors, sometimes referred to as cascaded switching or two-switch cascaded fabric, is for single-vendor fabrics only. 1 68 64 concurrent I/O operations are supported with FICON Express2 and FICON Express4 on System z servers. IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm Cascaded support is important for disaster recovery and business continuity solutions (see Figure 5-11). It can provide high availability connectivity, as well as the potential for fiber infrastructure cost savings for extended storage networks. FICON two-Director cascaded technology can allow for shared links, and therefore, improved utilization of intersite connected resources and infrastructure. Solutions such as GDPS® can benefit from the reduced inter-site configuration complexity that FICON support of cascaded Directors provides. Site A Site B Servers Servers Cascaded FICON Directors FC Fabric Switch FICON Channels Switch Cascaded FICON Directors FC Fabric Switch Switch Storage Devices FICON Channels Storage Device Figure 5-11 Two-site FICON connectivity with cascaded Directors While specific cost savings vary depending upon infrastructure, workloads, and size of data transfers, generally customers who have data centers separated between two sites can reduce the number of cross-site connections by using cascaded Directors. Further savings can be realized in the reduction of the number of channels and switch ports. Another important value of FICON support of cascaded Directors is its ability to provide high integrity data paths. The high integrity function is an integral component of the FICON architecture when configuring FICON channel paths through a cascaded fabric. To support the introduction of FICON cascaded switching, IBM has worked with Fibre Channel Director vendors to help ensure that robustness in the channel-to-control unit path is maintained to the same high standard of error detection, recovery, and data integrity that has existed for many years with both ESCON and the initial implementation of FICON. End-to-end data integrity is designed to be maintained through the cascaded Director fabric. Data integrity helps ensure that any changes to the customer’s data streams are always detected, and the data frames (data streams) are delivered to the correct endpoint, an endpoint being a FICON channel port or a FICON Control Unit port. For FICON channels, Cyclical Redundancy Checking (CRC) and Longitudinal Redundancy Checking (LRC) are bit patterns added to the data streams to allow for detection of any bit changes in the data Chapter 5. Fibre Connection 69 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm stream. With FICON support of cascaded switching, integrity features are introduced within the FICON channel and the FICON cascaded switch fabric to help ensure the detection and reporting of any mis-cabling actions occurring within the fabric during operational use that may cause a frame to be delivered to the wrong endpoint. A FICON channel, when configured to operate with a cascaded switch fabric, requires that the switch fabric supports high integrity. During initialization, the FICON channel will query the switch fabric to determine that it supports high integrity. If it does, then the channel will complete the initialization process, allowing the channel to operate with the fabric. Once a FICON switched fabric has been customized to support FICON cascaded Directors, and the required World-Wide Node Name (WWNN) and Domain IDs have been added in the fabric membership list, the Director will check that its Inter-Switch Links (ISLs) are attached to the correct Director before they are made operational. If an accidental cable swap occurs, the Director invokes logical path testing, reporting, isolation, and recovery. The high integrity fabric feature for cascaded FICON Directors protects against mis-cabling and misdirecting of data streams, as shown in Figure 5-12. The checking process is: 1. Channel initialization completes. 2. At some later time, miscabling occurs (for example, cables are swapped at a patch panel). 3. The director port enters invalid attachment state and notifies state change back to the System z server. 4. The System z server invokes the channel logical path testing, reporting, isolation, and error recovery. 5. Any I/O requests to the invalid route are discarded until the error is corrected. 6. Data is protected. Channel initialization completes. 5 3 4 ! FICON Director 1 FICON Director oops System z servers 6 Storage Devices 2 FICON Director Figure 5-12 High integrity fabric feature 70 IBM System z Connectivity Handbook FICON Director 6 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm High integrity fabric architecture support includes: 򐂰 Fabric binding support This is the ability of the fabric to prevent a switch from being added to the fabric that is not configured to support the “high integrity fabric”; for example, all switches in the required fabric must be defined to the fabric via a fabric membership list. 򐂰 Insistent domain IDs support This support will not allow a switch address to be automatically changed when a duplicate switch address is added to the enterprise fabric. It requires operator action to change a switch address. FICON control units and devices The control unit receives commands from the channel, receives data from and transmits data to the channel, and controls execution of commands and data transfer at associated devices. The control unit attaches to one or multiple links through port interfaces. Like the ESCON I/O architecture, FICON does not allow multiple physical control units (or control unit link interfaces) to reside on the same link. A control unit can contain multiple images with dedicated or shared physical control unit facilities. The FICON I/O architecture provides addressing for these multiple CU images. For ease in migration, control units can have both FICON and ESCON interfaces. A control unit can be attached through FICON (FC mode) and ESCON (CNC mode) channel paths from the same server using any topology combinations. Figure 5-13 shows that from one system, a single control unit can be attached through a mix of channel path types and topologies. Any ESCON topologies (CNC point-to-point and switched point-to-point), FICON types (FCV and FC) and topologies (FC point-to-point and switched point-to-point) can be combined to ease migration. FICON bridge port FCV LP 1 LP 2 MIF CNC FC Link ESCON Link CNC FC ESC ON Link ESCON Lin k CU ESCON Link FC Links FC LP 3 ESON Director FC Link FC Switch FC Link Figure 5-13 Mixing channel path types to one control unit A configuration of mixed channels is not recommended for the long term. Nevertheless, knowing all the supported combinations (CNC, FCV, FC) from one system to one control unit may help during migration phases. Chapter 5. Fibre Connection 71 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm Channel device addressing support An ESCON channel operating in CNC mode supports 1024 device addresses. This support is increased to 16 K devices (16384) for a FICON Bridge (FCV) channel and for a FICON native (FC) channel. Multiple Image Facility (MIF) MIF enables FICON channel sharing among Logical Partitions running on System z servers. See Chapter 2, “Channel Subsystem overview” on page 13. Spanned channels Spanning is the ability to configure channels to multiple Channel Subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the Channel Subsystem to which the logical partition is configured. FICON channels can be spanned across multiple CSSs in z196, z10 and z9 servers. See 2.1.2, “Multiple CSS concept” on page 14, for more details on MIF and spanned channels. Control Unit Port (CUP) The CUP function allows z/OS to manage a FICON Director with the same level of control and security as it is known from an ESCON switch. Host communication includes control functions like blocking/unblocking ports, as well as monitoring and error reporting functions. IBM Tivoli® System Automation for z/OS (SA for z/OS) includes support for FICON channels and FICON Directors. You can find additional information, updates, extensions, tools, and demonstrations for SA for z/OS at the following URL: http://www.ibm.com/software/tivoli/products/system-automation-390 Before using SA for z/OS in your FICON environment, check the latest maintenance recommendations in the appropriate z/OS subset of the 2817DEVICE, 2098DEVICE, 2097DEVICE, 2096DEVICE, and 2094DEVICE, Preventative Service Planning (PSP) buckets prior to implementation. z/OS discovery and auto-configuration (zDAC) z/OS discovery and auto-configuration (zDAC) is a function exclusive to z196 to automatically perform a number of I/O configuration definition tasks for new and changed disk and tape controllers that are connected to a FICON Director. It is designed to help simplify I/O configuration of z196 servers running z/OS and to help reduce complexity and setup time. The zDAC function is integrated into the existing Hardware Configuration Definition (HCD) tool. A policy can be defined in HCD according to the availability and bandwidth requirements, including parallel access volume (PAV) definitions, control unit numbers, and device number ranges. The zDAC proposed configurations is created as work I/O definition files (IODF) that can be converted to production IODF and activated. zDAC provide real-time discovery for the FICON fabric, subsystem and I/O device resource changes from z/OS. By exploring the discovered control units for defined logical control units (LCU) and devices, zDAC compares the discovered controller information with the current system configuration to determine delta changes to the configuration for a proposed configuration. All new added or changed logical control units and devices will be added into the proposed configuration, with proposed control unit and device numbers, and channel paths base on the defined policy. zDAC use channel path chosen algorithm to minimize single point of failure. 72 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm zDAC applies to all FICON features supported on z196 when configured as CHPID type FC. 5.3 Connectivity The connectivity options for the FICON I/O interface environment are discussed in this section. The maximum number of FICON channels supported based on each server can be found in Table 1-2 on page 10. Note: FICON Express4, FICON Express2, and FICON Express features are withdrawn from marketing. Only 2-port FICON Express4 features (feature codes 3318 and 3323) are offered for the System z10 BC server. Table 5-1 shows the available FICON features and their respective specifications. Details for each feature follow the table. Table 5-1 FICON feature specifications Feature code Feature name Connector type Cable type SM 9 µm 2319 2320 FICON Express LXc 3320 c LC Duplex FICON Express SX 3318 FICON Express2 LXc FICON Express2 SX Link data rate Server 1 or 2 Gbpsb z10 EC z10 BC z9 EC z9 BC 1 Gbps z10 EC z10 BC z9 EC z9 BC 300 m (984 feet) 1 or 2 Gbpsb z10 EC z10 BC z9 EC z9 BC kmg a 1 or 2 Gbpsb z10 EC z10 BC z9 EC z9 BC 1 Gbps z10 EC z10 BC z9 EC z9 BC 1 or 2 Gbpsb z10 EC z10 BC z9 EC z9 BC 1, 2, or 4 Gbpsb z10 BC z9 BC 10 km, 20 kma LC Duplex with MCP: MM 50 or MM 62.5 550 m (1804 feet) MM 62.5 µm 120 m (394 feet)d MM 50 µm SM 9 µm 3319 Unrepeated max. distance c FICON Express4-2C SX d 10 km, 20 LC Duplex LC Duplex with MCP: MM 50 or MM 62.5 550 m (1804 feet) MM 62.5 µm 120 m (394 feet)d d MM 50 µm 300 m (984 feet) MM 62.5 µm 55 m (180 feet)e at 160 MHz-km 70 m (230 feet)e at 200 MHz-km LC Duplex e MM 50 µm 150 m (492 feet) at 500 MHz-km 270 m (886 feet)e at 2000 MHz-km Chapter 5. Fibre Connection 73 5444ch05.fm Feature code Draft Document for Review July 21, 2010 6:06 pm Feature name Connector type Cable type SM 9 µm 3321 FICON Express4 10KM LX LC Duplex 3323 FICON Express4 SX FICON Express4-2C 4KM LX 550 m (1804 feet) LC Duplex 3325 FICON Express8 10KM LX LC Duplex 3326 FICON Express8 SX e MM 50 µm 150 m (492 feet) at 500 MHz-km 270 m (886 feet)e at 2000 MHz-km SM 9 µm 4 km LC Duplex FICON Express4 4KM LX f 55 m (180 feet)e at 160 MHz-km 70 m (230 feet)e at 200 MHz-km LC Duplex 3324 f SM 9 µm with MCP 550 m (1804 feet) SM 9 µm 4 km SM 9 µm with MCP 550 m (1804 feet)f SM 9 µm 10 km MM 62.5 µm 21 m (69 feet)g at 200 MHz-km MM 50 µm 50 m (164 feet)g at 500 MHz-km 150 m (492 feet)g at 2000 MHz-km LC Duplex Link data rate Server 1, 2, or 4 Gbpsb z196c z10 EC z10 BC z9 EC z9 BC 1, 2, or 4 Gbpsb z196c z10 EC z10 BC z9 EC z9 BC 1, 2, or 4 Gbpsb z10 BC z9 BC 1, 2, or 4 Gbpsb z196c z10 EC z10 BC z9 EC z9 BC 2, 4, or 8 Gbpsb z196 z10 EC z10 BC 2, 4, or 8 Gbpsb z196 z10 EC z10 BC 10 km SM 9 µm with MCP MM 62.5 µm 3322 Unrepeated max. distance a. At 2 Gbps, a maximum unrepeated distance of 12 km is supported. b. Supports auto-negotiate with neighbor node. c. Only supported on an upgrade. d. Maximum unrepeated distance at 2 Gbps. e. Maximum unrepeated distance at 4 Gbps. f. Maximum unrepeated distance at 1 Gbps. g. Maximum unrepeated distance at 8 Gbps. Note: IBM has qualified the 50 micron multimode 2000 MHz-km ISO/IEC OM3, TIA 850 nanometer laser-optimized 50/125 micrometer fiber optic cable for use when attaching System z to servers, switches/directors, disks, tapes, and printers. Support of the OM3 cable is designed to help facilitate use of the industry-standard fiber optic cabling when the unrepeated distances offered by 50 micron 500 MHz-km multimode fiber optic cabling are insufficient for data center requirements. The channels residing on a single FICON Express8, FICON Express4, FICON Express2, or FICON Express feature can be configured individually, and can be defined in different channel modes (FC, FCV, and FCP). FCV mode on System z is only supported with the FICON Express LX feature. 74 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm Note: FICON Express4 features are the last FICON features to support 1 Gbps link data rates. For information about extended distances for FICON, see Chapter 11, “Extended distance solutions” on page 167. 5.3.1 FICON Express8 The FICON Express8 features are exclusive to z196 and z10, and are designed to deliver increased performance compared with the FICON Express4 features. All FICON Express8 features use Small Form Factor Pluggable optics to permit each channel to be individually serviced in the event of a fiber optic module failure. The traffic on the other channels on the same feature can continue to flow if a channel requires servicing. The FICON Express8 features are ordered in 4-channel increments and are designed to be added concurrently. This concurrent update capability allows you to continue to run workloads through other channels while the FICON Express8 features are being added. FICON Express8 CHPIDs can be defined as a spanned channel and can be shared among Logical Partitions within and across CSSs. The FICON Express8 features are designed for connectivity to servers, switches, Directors, disks, tapes, and printers and can be defined as: 򐂰 CHPID type FC – Native FICON, High Performance FICON for System z (zHPF), and FICON Channel-to-Channel (CTC) traffic – Supported in the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments 򐂰 CHPID type FCP – Fibre Channel Protocol traffic for communication with SCSI devices – Supported in z/VM, z/VSE, and Linux on System z environments FICON Express8 10KM LX Feature code 3325 is designed to support unrepeated distances up to 10 kilometers (6.2 miles) at 8 Gbps. Each channel supports 9/125 micrometer single mode fiber optic cable terminated with an LC Duplex connector. FICON Express8 SX Feature code 3326 is designed to support unrepeated distances up to 150 meters (492 feet) at 8 Gbps. Each channel supports a 50/125 micrometer multimode fiber optic cable or a 62.5/125 micrometer multimode fiber optic cable terminated with an LC Duplex connector. 5.3.2 FICON Express4 The FICON Express4 features have four (or two for the 2-port features) independent channels. Each feature occupies a single I/O slot, utilizing one CHPID per channel. Each channel supports 1 Gbps, 2 Gbps, and 4 Gbps link data rates with auto-negotiation. Chapter 5. Fibre Connection 75 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm All FICON Express4 features use Small Form Factor Pluggable optics to permit each channel to be individually serviced in the event of a fiber optic module failure. The traffic on the other channels on the same feature can continue to flow if a channel requires servicing. The FICON Express4 features are ordered in 4-channel (or 2-channel) increments and are designed to be added concurrently. This concurrent update capability allows you to continue to run workloads through other channels while the FICON Express4 features are being added. FICON Express4 CHPIDs can be defined as a spanned channel and can be shared among Logical Partitions within and across CSSs. The FICON Express4 features are designed for connectivity to servers, switches, Directors, disks, tapes, and printers and can be defined as: 򐂰 CHPID type FC – Native FICON, High Performance FICON for System z (zHPF), and FICON Channel-to-Channel (CTC) traffic – Supported in the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments 򐂰 CHPID type FCP – Fibre Channel Protocol traffic for communication with SCSI devices – Supported in z/VM, z/VSE, and Linux on System z environments Note: FICON Express4 is supported on z196 servers only if carried forward on an upgrade. FICON Express4 10KM LX Feature code 3321 is designed to support unrepeated distances up to 10 kilometers (6.2 miles) at 4 Gbps. Each channel supports 9 micron single mode fiber optic cable terminated with an LC Duplex connector. Multimode (62.5 or 50 micron) fiber cable can be used with the FICON Express4 10KM LX feature. The use of multimode cable types requires a mode conditioning patch (MCP) cable. This feature should be used when the unrepeated distance between devices is greater than 4 kilometers or the link loss budget between devices exceeds 2 dB. A 10KM LX transceiver is designed to interoperate with a 10KM LX transceiver. Interoperability of 10 km transceivers with 4 km transceivers is supported, provided that the unrepeated distance does not exceed 4 km. FICON Express4 4KM LX Feature code 3324 is designed to support unrepeated distances up to 4 kilometers (2.5 miles) at 4 Gbps. Interoperability of 4 km transceivers with 10 km transceivers is supported, provided that the unrepeated distance does not exceed 4 km. Each channel supports 9 micron single mode fiber optic cable terminated with an LC Duplex connector. The multimode (62.5 or 50 micron) fiber cable can be used with the FICON Express4 4 KM LX feature. The use of multimode cable types requires a mode conditioning patch (MCP) cable. 76 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm FICON Express4-2C 4KM LX Feature code 3323, with two channels per feature, is designed to support unrepeated distances up to 4 kilometers (2.5 miles) at 4 Gbps. Interoperability of 4 km transceivers with 10 km transceivers is supported, provided that the unrepeated distance does not exceed 4 km. Note: FICON Express4-2C 4KM LX is only available on System z10 BC and System z9 BC servers. Each channel supports 9 micron single mode fiber optic cable terminated with an LC Duplex connector. A multimode (62.5 or 50 micron) fiber cable can be used with the FICON Express4-2C 4 KM LX feature. The use of multimode cable types requires a mode conditioning patch) cable. FICON Express4 SX Feature code 3322 is designed to support unrepeated distances up to 380 meters (1247 feet) at 4 Gbps. Each channel supports a 62.5 micron or 50 micron multimode fiber optic cable terminated with an LC Duplex connector. FICON Express4-2C SX Feature code 3318, with two channels per feature, is designed to support unrepeated distances up to 380 meters (1247 feet) at 4 Gbps. Note: FICON Express4-2C SX is only available on System z10 BC and System z9 BC servers. Each channel supports a 62.5 micron or 50 micron multimode fiber optic cable terminated with an LC Duplex connector. 5.3.3 FICON Express2 The FICON Express2 SX and FICON Express2 LX features have four independent channels, each feature occupying a single I/O slot, utilizing one CHPID per channel, four CHPIDs per feature, while continuing to support 1 Gbps and 2 Gbps link data rates. The link speed is auto-negotiated. The FICON Express2 SX and LX features are ordered in 4-channel increments and designed to be added concurrently. This concurrent update capability allows you to continue to run workloads through other channels while the FICON Express2 features are being added. FICON Express2 CHPIDs can be defined as a spanned channel and can be shared among Logical Partitions within and across CSSs. The FICON Express2 features are designed for connectivity to servers, switches, Directors, disks, tapes, and printers and can be defined as: 򐂰 CHPID type FC – Native FICON, High Performance FICON for System z (zHPF), and FICON Channel-to-Channel (CTC) traffic Chapter 5. Fibre Connection 77 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm – Supported in the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments 򐂰 CHPID type FCP – Fibre Channel Protocol traffic for communication with SCSI devices – Supported in z/VM, z/VSE, and Linux on System z environments Note: FICON Express2 is supported on System z10 and System z9 servers only if carried forward on an upgrade. FICON Express2 LX Feature code 3319 is designed to support unrepeated distances up to 10 kilometers (6.2 miles) at 2 Gbps. Each channel supports a 9 micron single mode fiber optic cable terminated with an LC Duplex connector. A multimode (62.5 or 50 micron) fiber cable can be used with the FICON Express2 LX feature. The use of multimode cable types requires a mode conditioning patch (MCP) cable. FICON Express2 SX Feature code 3320 is designed to support unrepeated distances up to 500 meters (1640 feet) at 2 Gbps. Each channel supports a 62.5 micron or 50 micron multimode fiber optic cable terminated with an LC Duplex connector. 5.3.4 FICON Express The two channels residing on a single FICON Express feature occupy one I/O slot in the System z I/O cage. Each channel can be configured individually and can support 1 or 2 Gbps data rates. The FICON Express features are designed for connectivity to servers, switches, Directors, disks, tapes, and printers and can be defined as: 򐂰 CHPID type FC – Native FICON and FICON Channel-to-Channel (CTC) traffic – Supported in the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments 򐂰 CHPID type FCP – Fibre Channel Protocol traffic for communication with SCSI devices – Supported in z/VM, z/VSE, and Linux on System z environments 򐂰 CHPID type FCV (FICON Express LX only) – Supported in the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments Note: FICON Express is supported on System z10 and System z9 servers only if carried forward on an upgrade. FICON Express LX Feature code 2319 is designed to support unrepeated distances up to 10 kilometers (6.2 miles) at 1 Gbps. Each channel supports a 9 micron single mode fiber optic cable terminated with an LC Duplex connector. 78 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm A multimode (62.5 or 50 micron) fiber cable can be used with the FICON Express LX feature. The use of multimode cable types requires a mode conditioning patch (MCP) cable. FICON Express SX Feature code 2320 is designed to support unrepeated distances up to 860 meters (2822 feet) at 1 Gbps. Each channel supports 62.5 micron or 50 micron multimode fiber optic cable terminated with an LC Duplex connector. 5.3.5 Qualified FICON and FCP products For information regarding System z qualified FICON and Fibre Channel Protocol (FCP) products, as well as products that support intermixing of FICON and FCP within the same physical FC switch or FICON Director, go to: http://www.ibm.com/systems/z/connectivity On the left, click "Library" and locate the listing of "Hardware products for servers" in the middle of the Web page. Then click “Switches and directors qualified for IBM System z FICON and FCP channels”. 5.3.6 FICON advantages over ESCON FICON has the following advantages over ESCON: 򐂰 FICON improves distance solutions supporting disaster recovery applications by a factor of 10. – ESCON data droop is significant at 9 km. – FICON has negligible data droop up to 100 km. 򐂰 FICON provides up to twenty times greater bandwidth per channel. – Maximum ESCON data rate is 17 MBps (20 MBps link data rate). – Maximum FICON data rate is >730 MBps (800 MBps link data rate). 򐂰 FICON supports 16 times as many devices. – ESCON supports 1 K unit addresses per channel. – FICON supports 16 K unit addresses per channel. 򐂰 FICON uses fiber more efficiently. – ESCON has half-duplex data transfer. – FICON has full-duplex data transfer. 򐂰 FICON provides greater configuration flexibility. – ESCON is circuit-switched. There are restrictions on large and small block intermixing. – FICON is packet-switched. There are no restrictions on large and small block intermixing. 򐂰 FICON provides relief for “channel constrained” systems. – FICON Express8 features delivers up to 2688 ESCON equivalent channels on z196 and z10 EC, with the maximum number of 336 FICON channels. – FICON can span across multiple CSSs, while ESCON can only be shared within a CSS. Additional FICON advantages include: Chapter 5. Fibre Connection 79 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 Protection of customer’s ESCON control unit investments, reducing cable bulk and extending the maximum distance 򐂰 Reduced IOQ (I/O queue time) More Parallel Access Volumes possibilities because FICON supports attachment of up to 16384 devices, reducing the time to wait at a UCB. 򐂰 Reduced Pending time – Channel Busy conditions are reduced by FICON’s ability to perform multiple concurrent I/O operations on the same channel path. – Destination Port Busy conditions are reduced because the FICON Switch uses switch port buffer credits (the control unit FC link is able to perform several concurrent I/O operations). 򐂰 Reduced Connect time Connect time is reduced for long records. There is more data per FICON frame (packet). 򐂰 Reduced Disconnect time Reduced Reconnect Destination Port Busy. 5.3.7 Software support Operating systems that support FICON on System z servers include: 򐂰 z/OS All in-service releases of z/OS; for more information go to: http://www.ibm.com/systems/z/os/zos/support/index.html 򐂰 z/VM All in-service releases of z/VM; for more information go to: http://www.vm.ibm.com/techinfo/lpmigr/vmleos.html 򐂰 z/VSE All in-service releases of z/VSE; for more information go to: http://www.ibm.com/servers/eserver/zseries/os/vse/support/support.htm 򐂰 z/Transaction Processing Facility (z/TPF): All in-service releases of z/TPF; for more information go to: http://www.ibm.com/software/htp/tpf/index.html 򐂰 Linux For details about Linux on System z support and downloads, go to: http://www.ibm.com/servers/eserver/zseries/os/linux/dist.html Note: Certain functionality may require specific levels of an operating system, or PTFs, or both; that information is provide when appropriate within this chapter. 5.3.8 Resource Measurement Facility (RMF) RMF reporting is available for all FICON features. This enables you to capture performance data through the following: 򐂰 Channel Path Activity report (of primary interest) 򐂰 Device Activity report 80 IBM System z Connectivity Handbook 5444ch05.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 FICON Director Activity report 򐂰 I/O Queuing Activity report With these reports, you can analyze possible bandwidth bottlenecks and perform root cause analysis. For further performance information refer to the System z I/O connectivity Web page at: http://www.ibm.com/systems/z/hardware/connectivity/ficon_performance.html 5.4 References The following publications contain information related to the topics covered in this chapter: 򐂰 FICON I/O Interface Physical Layer, SA24-7172 򐂰 IBM Cabling System Optical Fiber Planning and Installation, GA27-3943 򐂰 Linux on System z: Fibre Channel Protocol Implementation Guide, SG24-6344 򐂰 FICON Planning and Implementation Guide, SG24-6497 For the Fibre Channel Standard publications see: http://www.t11.org For information about the SCSI Storage Interface standards see: http://www.t10.org/ Chapter 5. Fibre Connection 81 5444ch05.fm 82 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444ch06.fm 6 Chapter 6. Channel-to-channel This chapter describes channel-to-channel (CTC) connections for the zEnterprise 196, System z10, and System z9 servers, using ESCON and FICON channels to interconnect different servers and logical partitions. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 83 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm 6.1 Description The channel-to-channel function simulates an input/output (I/O) device that can be used by one system control program to communicate with another system control program. It provides the data path and synchronization for data transfer between two channels. When the CTC option is used to connect two channels that are associated with different systems, a loosely coupled multiprocessing system is established. The CTC connection, as viewed by either of the channels it connects, has the appearance of an unshared input/output device. The CTC is selected and responds in the same manner as any I/O device. It differs from other I/O devices in that it uses commands to open a path between the two channels it connects, and then synchronizes the operations performed between the two channels. CTC support exists for: 򐂰 ESCON channels 򐂰 FICON channels via ESCON Director with the FICON Bridge feature 򐂰 FICON Native CTC (FCTC) CTC control units and associated devices provide the infrastructure for intersystem communication. Exploiters of CTC communication include, but are not limited to: 򐂰 Cross-System Coupling Facility (XCF) pathin and pathout devices for sysplex intersystem communication (z/OS). For small message sizes, CTC may be more efficient than passing messages via the Coupling Facility (CF). 򐂰 VTAM® Multi-Path Channel (MPC) read/write devices. 򐂰 TCP/IP read/write devices. 򐂰 IMS™ Multiple Systems Coupling (MCS) read/write devices. 򐂰 Remote Spooling Communications Subsystem (RSCS). 򐂰 VM Pass-Through Facility. The ESCON channel-to-channel (with CHPID types of CTC for control unit and CNC for channel) is an HCD/IOCP configuration option of an ESCON-capable server. The CTC option is specified in the IOCP configuration, which results in the CTC LIC being loaded into the ESCON channel hardware at Power-on Reset (POR). System z servers running z/OS or z/VM allow dynamic I/O configuration using the Hardware Configuration Definition (HCD). ESCON channels that operate in CTC mode (also called Serial CTC or SCTC) support both extended mode and Basic mode operations. The FICON channel in FICON native (CHPID type FC) mode on the System z servers provides enhanced CTC control unit support, with increased connectivity and more flexible channel usage. 84 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch06.fm 6.2 Connectivity CTC communication is used in the following configurations: 򐂰 Loosely coupled A loosely coupled configuration has more than one Central Processing Complex (CPC), sharing DASD but not main storage. The CPCs are connected by channel-to-channel communications and are managed by more than one z/OS image. 򐂰 Base Sysplex In the base sysplex configuration, CPCs are connected by CTC communications and a shared data set to support the communication. When more than one CPC is involved, Server Time protocol (STP) or a Sysplex Timer® synchronizes the time on all systems. 򐂰 Parallel Sysplex The Coupling Facility enables high-performance, multisystem data sharing across all systems. The CTCs in Parallel Sysplex are used to support cross-system Coupling Facility (XCF) signalling paths between systems in the sysplex. For information about distance support for CTC channels, see Chapter 11, “Extended distance solutions” on page 167. 6.2.1 ESCON CTC channel connectivity In the Enterprise Systems CONnection (ESCON) I/O interface environment, a stand-alone CTC control unit is no longer used to provide CTC adapter and switching functions. The CTC adapter function is provided by an ESCON channel defined as a CTC by the TYPE keyword in the IOCP CHPID statement. In an ESCON CTC channel configuration, there are always two channels involved for the communication: an ESCON channel defined to operate in CTC mode on one side of the connection, and an ESCON channel defined to operate in CNC mode on the other side. A CTC channel path communicates with a CNC channel, and conversely. The switching function is performed by the ESCON director. It allows one CTC ESCON channel to communicate with several ESCON CNC channels, that is, with other servers or logical partitions. The ESCON CTC connection can either be point-to-point or switched point-to-point. Chapter 6. Channel-to-channel 85 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm Figure 6-1 illustrates ESCON CTC channel communication, with several ESCON CNC channels using an ESCON director. ESCON CTC channel ESCON CNC channel ESCON CNC channel ESCON Director ESCON CNC channel Figure 6-1 Using ESCON Director for CTC switching ESCON CTC channel structure An ESCON CTC function is implemented and integrated in its Licensed Internal Code (LIC). An ESCON CTC channel can be logically divided into two sections: 򐂰 Channel section: Performs the regular channel functions. 򐂰 CTC control unit section: Synchronizes the operations performed between two channels. See ESA/390 ESCON Channel-to-Channel Adapter, SA22-7203, for more information. The ESCON CTC channel and the ESCON CNC (a mnemonic for an ESCON connection to an ESCON-capable device) channel use the same ESCON channel hardware, but different LIC. A virtual link exists between the channel section and the CTC control unit section. Logically, a CTC control unit exists between its own channel section and the ESCON CNC channel it is connected to by the ESCON I/O interface. The CTC channel acts like a control unit, not as a channel, on the ESCON I/O interface to the connected CNC channels. See Figure 6-2. ESCON CTC channel Channel section CTC CU section ESCON I/O ESCON CNC channel interface Figure 6-2 ESCON CTC channel structure Shared/unshared channel ESCON channels can be defined as dedicated, reconfigurable, or shared. Channels in a server complex without Multiple Image Facility (MIF) capability cannot be shared by logical partitions. Unshared channels are dedicated to a single partition. If an 86 IBM System z Connectivity Handbook 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm unshared channel is reconfigurable, it can be reconfigured from one partition to another partition. MIF allows logical partitions to share channel paths, and so optionally they can share any control units and associated I/O devices configured to these shared channels. With MIF, logical partitions can share channel paths to reduce the number of physical connections between servers. Both CTC and CNC channels can be defined as shared channels; see Figure 6-3. LP1B LP1A LP2A M I F CTC Channel LP2A CNC Channel M I F LP2B LP2B Figure 6-3 CTC connection with shared channels 6.2.2 FICON (FCV)-to-ESCON CTC connectivity A FICON channel (in FCV mode) can be used in the same way as an ESCON CNC channel to participate in ESCON CTC connections and communication. The CTC control unit function is provided by the ESCON CTC channel. A FICON FCV channel can intermix concurrent connections and I/O operations between ESCON CTCs and other ESCON control units (for example, DASD control units). The communication between a FICON (FCV mode) channel and an ESCON CTC channel is always a switched point-to-point connection. A FICON channel (in FCV mode) can concurrently connect from one FICON FCV to up to 8 ESCON CTC channels. Chapter 6. Channel-to-channel 87 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm Figure 6-4 illustrates ESCON CTC channel communications to FICON (FCV) and ESCON CNC channels. System A FICON Link FCV System B FCV CTC CTC ESCON Director bridge port ESCON Links System C CNC DASD CU TAPE CU ESCON Links CTC Figure 6-4 ESCON CTC channel configuration with FICON FCV and CNC channels The channel systems are: 򐂰 CTC channel system A and CNC channel system C can form a CTC connection. 򐂰 CTC channel system A and FCV channel system B can form a CTC connection. 򐂰 CNC channel system C and FCV channel system B cannot form a CTC connection. 򐂰 CNC channel system C and FCV channel system B can access other ESCON I/O control units (for example, ESCON DASD and ESCON tape). 6.2.3 FICON channel-to-channel (FCTC) connectivity Channel-to-channel communication in a FICON environment is provided between two FICON native (CHPID type FC) channel FCTC control units, with at least one of the two FCTC CUs being defined on a FICON native channel on a System z. A FICON CTC configuration can be implemented in different ways. However, the following considerations apply to all FICON CTC configurations: 򐂰 The server at each end of a FICON CTC connection uses a FICON native (CHPID type FC) channel. 򐂰 The FICON native channel at each end of the CTC connection has a FICON CTC control unit defined. An FCTC control unit can be defined on a FICON native channel on System z. 򐂰 The FICON CTC control unit is defined as type FCTC. The FICON CTC devices on the FCTC control unit are defined as type FCTC. 88 IBM System z Connectivity Handbook 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 The FCTC control function on the System z server can communicate with an FCTC control unit defined on a FICON native channel on any server supporting FICON. This is covered in the next section. 򐂰 The FICON native channel at each end of the FICON CTC connection, supporting the FCTC control units, can also communicate with other FICON native control units, such as disk and tape. In a FICON CTC configuration, although FICON CTC control units are defined at each end, only one end provides the FICON CTC control unit function. During initialization of the logical connection between two ends of a FICON CTC connection, the channel that will provide the FICON CTC control unit function is determined using an algorithm that, where possible, results in balancing of the number of FCTC CU functions that each end of the logical connection is providing. The algorithm uses the channel with the lower FC World Wide Name (WWN) to provide the FICON CTC control unit function. Unlike the ESCON channel CTC communication, which uses a pair of ESCON CTC-CNC channels, the FICON native channel CTC communication does not require a pair of channels because it can communicate with any FICON native channel that has a corresponding FCTC control unit defined. This means that FICON CTC communications can be provided using only a single FICON native channel per server. Single FICON FCTC CU function channel on one server A single FICON native channel connected to a FICON Director can provide the FICON CTC communications between the operating systems in the logical partitions on a single server. It can also be used to communicate to other I/O control units. For the FCTC function in this configuration, the Destination Port Address is the same as the Source Port Address. See Figure 6-5. Server 1 LP1 LP2 LP3 FC FICON Director FICON Channel FICON native (FC) Mode Disk Control Unit Figure 6-5 Single FICON channel on one server Chapter 6. Channel-to-channel 89 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm Single FICON channel between two or more servers A single FICON native channel with FICON control units defined can be used to communicate between the operating systems in the Logical Partitions on the same server as well as Logical Partitions on other servers. It can also be used to communicate to other I/O control units. The FC channels are connected to a FICON Director. See Figure 6-6. Server 1 LP1 Server 2 LP2 LP3 FC FC LPA FICON Director FICON Channel FICON native (FC) Mode LPB FC LPC FC FICON Channel FICON native (FC) Mode Disk Control Unit Figure 6-6 Single FICON channel per server Two FICON channels on one server Although a single FICON native channel per server can provide CTC connections across multiple servers, for large FICON configurations we recommend using at least one pair of FC channels. Using a pair of FC channels allows the installation to maintain the same CTC device definition methodology for FICON as was previously used for ESCON. The FICON channels can also support the definition of FCTC control units as well as other I/O control units, such as disk and tape, at the same time. A sample configuration with two FICON native channels providing FCTC-to-FCTC communications is shown in Figure 6-7. A FICON native channel supports up to 255 logical control units (LCUs) and up to 16K unit addresses per channel. Since the FICON native channel supports a larger number of devices, installations with a high number of logical partitions in an FCTC complex are easier to design and define. Server 1 LP1 Server 2 LP2 FC LP3 FC LPA FICON Director FICON Channel FICON native (FC) Mode Disk Control Unit Figure 6-7 Two FICON channels between two servers 90 IBM System z Connectivity Handbook FC LPB LPC FC FICON Channel FICON native (FC) Mode 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm FICON FCTC support via cascaded Directors FICON native channels on System z servers support cascaded Fibre Channel Directors. This support is for a two-Director configuration only. With cascading, a FICON native channel, including the FICON channel-to-channel (FCTC) function, can connect a server to a device or another server via cascaded FICON Directors. This cascaded Director support, delivered in conjunction with IBM Fibre Channel Director partners, is for all FICON channels implemented on FICON Express, FICON Express2, FICON Express4, and FICON Expres8 features. See “FICON support of cascaded Directors” on page 68. Differences between ESCON and FICON CTC There are several differences between the ESCON and FICON CTC implementations, as shown in Table 6-1. Table 6-1 ESCON and FICON CTC differences Characteristic ESCON FICON Number of required channels At least 2 1 or 2 Channel dedicated to CTC function Yesa No Number of unit addresses supported Up to 512 Up to 16384 Data transfer bandwidth 12 - 17 MBps Up to 340+ MBpsb Number of concurrent I/O operations 1 Up to 32c Data transfer mode Half duplex Full duplex a. CNC CHPID types do not need to be dedicated to the CTC function b. System z9 server using a FICON Express4 channel c. Up to 64 with FICON Express8, FICON Express4, or FICON Express2 features The details of these differences are as follows: 򐂰 ESCON CTC connectivity is provided by a pair of ESCON channels, one defined as CTC and the other defined as CNC. At least two ESCON channels are required. FICON CTC connectivity can be implemented using one or two FICON native channels. 򐂰 An ESCON channel defined as CTC can only support the CTC function. Only an SCTC control unit can be defined on an ESCON CTC channel. The FICON native channel supporting the FCTC control unit can communicate with an FCTC control unit on either System z, and at the same time the same FICON channel can also support operations to other I/O control unit types such as disk and tape. 򐂰 An ESCON CTC channel supports a maximum of 512 unit addresses (devices). A FICON native channel supports a maximum of 16,384 unit addresses (devices). 򐂰 An ESCON channel has a data transfer bandwidth of 12 to 17 MB, significantly less than the 60 to 800 MB of a FICON channel. 򐂰 An ESCON channel supports only one actively communicating I/O operation at a time, whereas the FICON channel supports up to 64 concurrent I/O operations. 򐂰 An ESCON channel operates in half-duplex mode, transferring data in one direction only at any given time. A FICON channel operates in full duplex mode, sending and receiving data at the same time. Chapter 6. Channel-to-channel 91 5444ch06.fm Draft Document for Review July 21, 2010 6:06 pm 6.3 References The following publications contain information related to the topics discussed in this chapter: 򐂰 lnput/Output Configuration Program User’s Guide for ICP IOCP, SB10-7037 򐂰 System z ESCON and FICON Channel-to-Channel Reference, SB10-7034 򐂰 Enterprise Systems Connection (ESCON) Implementation Guide, SG24-4662 򐂰 Parallel Sysplex Overview: Introducing Data Sharing and Parallelism in a Sysplex, GC28-1860 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 BC Technical Introduction, SG24-7241 򐂰 IBM System z10 Enterprise Class Technical Introduction, SG24-7515 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 FICON Planning and Implementation Guide, SG24-6497 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 92 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch08.fm 7 Chapter 7. Open Systems Adapter-Express This chapter describes the Open Systems Adapter-Express3 (OSA-Express3) and Open Systems Adapter-Express2 (OSA-Express2) features. These features provide connectivity to other servers and clients on 1000BASE-T Ethernet (10, 100, and 1000 Mbps), Gigabit Ethernet (GbE), and 10 Gigabit Ethernet environments. Terminology: If not specifically stated otherwise, the term OSA applies to the OSA-Express3 and OSA-Express2 features throughout this book. The following topics are covered: 򐂰 Functional description 򐂰 OSA capabilities 򐂰 Connectivity © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 93 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.1 Functional description The Open Systems Adapter-Express3 (OSA-Express3), and OSA-Express2 features comprise a number of integrated hardware features that can be installed in a System z I/O cage, becoming integral components of the server’s I/O subsystems. They provide high function, connectivity, bandwidth, data throughput, network availability, reliability, and recovery. All OSA-Express3 and OSA-Express2 features are hot-pluggable. Figure 7-1 shows the OSA-Express3 and OSA-Express2 Ethernet features available on the zEnterprise 196, System z10 and System z9 servers. Figure 7-1 OSA-Express3 and OSA-Express2 Ethernet connectivity For a complete list and description of all the OSA features offered on the zEnterprise 196, System z10 and System z9 servers, go to Table 7-2 on page 119. 7.1.1 Operating modes The integration of a channel path with network ports makes the OSA a unique channel or CHPID type, recognized by the hardware I/O configuration on a port-by-port basis as one of the following: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 94 OSD (Queued Direct Input/Output) OSE (non Queued Direct Input/Output) OSC (OSA Integrated Console Controller) OSN (Open System Adapter for NCP) OSX (OSA-Express for zEnterprise BladeCenter Extension (zBX)) OSM (OSA-Express for management of an ensemble) IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Note that not all features support all CHPID types. Table 7-1 gives an overview of the type of traffic supported and whether OSA/SF is required to configure the OSA-Express3 or OSA-Express2 port, based on the supported modes of operation (CHPID types). Table 7-1 Supported CHPID types CHPID type Feature SNA/APPN/ HPR traffic TCP/IP traffic 3270 traffic OSA/SF OSD OSA-Express3 10GbE LR OSA-Express3 10GbE SR OSA-Express2 10GbE LR OSA-Express3 GbE OSA-Express2 GbE OSA-Express3 1000BASE-T OSA-Express2 1000BASE-T Noa,b Noa,b Noa,b Noa,b Noa,b Noa,b Noa,b Yes Yes Yes Yes Yes Yes Yes No No No No No No No Optional Optional Optional Optional Optional Optional Optional OSE OSA-Express3 1000BASE-T OSA-Express2 1000BASE-T Yes Yes Yes Yes No No Required Required OSC OSA-Express3 1000BASE-T OSA-Express2 1000BASE-T No No No No Yes Yes N/A N/A OSM OSA-Express3 1000BASE-T No Yes No N/A OSN OSA-Express3 GbE OSA-Express2 GbE OSA-Express3 1000BASE-T OSA-Express2 1000BASE-T Yesc Yesc Yesc Yesc No No No No No No No No Optional Optional Optional Optional OSX OSA-Express3 10GbE LR OSA-Express3 10GbE SR No No Yes Yes No No N/A N/A a. SNA over IP with the use of Enterprise Extender or TN3270 (see 7.2.17, “Enterprise Extender” on page 114 and 7.2.18, “TN3270E Server” on page 114). b. Layer 2 support allows for non-IP protocols, such as SNA (see 7.2.13, “Layer 2 support” on page 110). c. Supports SNA PU Type 5 and PU Type2.1 (see 7.2.19, “OSA for NCP support” on page 114). Open Systems Adapter Support Facility (OSA/SF) OSA/SF is a host-based tool used to customize and manage all OSA features. 򐂰 OSA/SF is not required for the OSA feature that is configured for the QDIO mode, or the default IP Passthru non-QDIO mode. However, it can be used for problem determination purposes. 򐂰 OSA/SF is not required for OSA port CHPID types OSC and OSN, although information about channel usage can by displayed through OSA/SF for OSN CHPIDs. 򐂰 OSA/SF is a required facility when the OSA feature is being configured for shared non-QDIO mode and where SNA definitions are involved. One OSA/SF application can communicate with all OSA features in a System z server. OSA/SF communicates with an OSA feature through a device (type OSAD) defined via HCD/IOCP. For more details, refer to 7.1.5, “OSA/SF support” on page 101. Chapter 7. Open Systems Adapter-Express 95 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm QDIO versus non-QDIO Figure 7-2 illustrates the much shorter I/O process when in QDIO mode compared with non-QDIO mode. I/O interrupts and I/O path-lengths are minimized, resulting in improved performance versus non-QDIO mode, reduction of System Assist Processor (SAP) utilization, improved response time, and server cycle reduction. Non-QDIO (LCS) Host Memory QDIO QDIO Host Memory Host Memory Store and forward OSA-Express2 Data router OSA-Express3 LAN LAN Channel Control Unit NIC OSA-Express LAN Figure 7-2 Non-QDIO data path versus QDIO data paths Note that OSA-Express3 features use Direct Memory Access (DMA) and a data router model to eliminate store and forward delays that could occur with the OSA-Express2 features when in QDIO mode. Also in QDIO mode, all OSA features receive configuration information from the host dynamically. This reduces configuration and setup time, eliminates duplicate data entry, and reduces the possibility of data entry errors and incompatible definitions. We recommend the use of QDIO mode wherever possible. 7.1.2 QDIO mode QDIO is a highly efficient data transfer mechanism that is designed to dramatically reduce system overhead and improve throughput by using system memory queues and a signaling protocol to directly exchange data between the OSA microprocessor and network software. QDIO is the interface between the operating system and the OSA hardware. The components that make up QDIO are DMA, data router (OSA-Express3 only), Priority Queuing (z/OS only), dynamic OSA Address Table building, LPAR-to-LPAR communication, and Internet Protocol (IP) Assist functions. QDIO supports IP and non-IP traffic with the OSA-Express3 and OSA-Express2 features. These features support two transport modes: Layer 2 (Link Layer) for IP and non-IP traffic, and Layer 3 (Network Layer) for IP traffic only. A more detailed discussion about the Layer 2 support is provided in 7.2.13, “Layer 2 support” on page 110. 96 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Direct Memory Access (DMA) OSA and the operating system share a common storage area for memory-to-memory communication, reducing system overhead and improving performance. Data can move directly from the OSA microprocessor to system memory and vice versa, utilizing a store and forward technique in DMA. There are no read or write channel programs for data exchange. For write processing, no I/O interrupts have to be handled. For read processing, the number of I/O interrupts is minimized. Data router With OSA-Express3, what was previously done in Licensed Internal Code (LIC) is now performed in hardware. There is additional logic in the IBM ASIC to handle packet construction, inspection, and routing, thereby allowing packets to flow between host memory and the LAN at line speed without firmware intervention. With the data router, the store and forward technique in DMA is no longer used, which enables a direct host memory-to-LAN flow and avoids a hop. It is designed to reduce latency and increase throughput for standard frames (1492 bytes) and jumbo frames (8992 bytes). Priority queuing Priority queuing is a capability supported by the QDIO architecture and introduced with the Service Policy Server (for z/OS environments only). It sorts outgoing IP message traffic according to the service policy you have set up for the specific priority assigned in the IP header. This is an alternative to the best effort priority assigned to all traffic in most TCP/IP networks. Priority queuing allows the definition of four different priority levels for TCP/IP traffic through the OSA features defined for QDIO. For example, you can grant interactive communications the highest priority while assigning batch traffic the lowest, with two additional categories in between, perhaps based on particular user groups or projects. QDIO uses four write (outbound) queues and one read (inbound) queue for each TCP/IP stack sharing the OSA feature. OSA signals to z/OS Communications Server when there is work to do. z/OS Communications Server puts outbound packets in one of the four queues, based on priority settings. At a certain time, z/OS Communications Server signals the OSA feature that there is work to do. The OSA feature searches the four possible outbound queues by priority and sends the packets to the network, giving more priority to queues 1 and 2, and less priority to queues 3 and 4. For example, if there is data on every queue, queue 1 is served first, then portions of queue 2, then fewer portions of queue 3, then even fewer portions of queue 4, and then back to queue 1. This means that if there were four transactions running across the four queues, over time queue 1 would finish first, queue 2 would finish second, and so on. Note: With OSA-Express3 and OSA-Express2, priority queuing is by default enabled; this reduces the total number of supported TCP/IP stacks and devices (see “Maximum TCP/IP stacks and subchannels” on page 100). Dynamic OSA Address Table (OAT) update With QDIO, this process simplifies installation and configuration tasks. The definition of IP addresses is done in one place, the TCP/IP profile, thus removing the requirement to enter the information into the OAT using the OSA Support Facility (OSA/SF). Chapter 7. Open Systems Adapter-Express 97 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm The OAT entries will be dynamically built when the corresponding IP device in the TCP/IP stack is started. At device activation, all IP addresses contained in the TCP/IP stack’s IP HOME list are downloaded to the OSA port, and corresponding entries are built in the OAT. Subsequent changes to these IP addresses will cause a corresponding update of the OAT. LPAR-to-LPAR communication Access to an OSA port can be shared among the system images that are running in the logical partitions to which the channel path is defined to be shared. Also, access to a port can be shared concurrently among TCP/IP stacks in the same logical partition or in different logical partitions. When port sharing, an OSA port operating in QDIO mode has the ability to send and receive IP traffic between logical partitions without sending the IP packets out to the LAN and then back to the destination logical partition. For outbound IP packets, the OSA port uses the next-hop IP address within the packet to determine where it is sent. If the next-hop IP address had been registered by another TCP/IP stack sharing the OSA port, then the packet will be sent directly to that TCP/IP stack, and not onto the LAN. This makes the forwarding of IP packets possible within the same host system. Internet Protocol Assist (IPA) functions OSA QDIO assists in IP processing and offloads the TCP/IP stack functions for the following: 򐂰 Multicast support (See “TCP/IP multicast and broadcast support” on page 107.) 򐂰 Broadcast filtering (See “TCP/IP multicast and broadcast support” on page 107.) 򐂰 Building MAC and LLC headers 򐂰 Address Resolution Protocol (ARP) processing (See “ARP cache management” on page 108.) 򐂰 Checksum offload (See “Checksum offload support for z/OS and Linux on System z” on page 109.) QDIO functions The following QDIO functions discussed in this section are supported on z196, z10, and z9 servers. TCP/IP functions The TCP/IP functions are: 򐂰 Large Send for TCP/IP traffic for OSA-Express3 and OSA-Express2 (See “Large send for TCP/IP traffic” on page 104.) 򐂰 640 TCP/IP stacks (See “Maximum TCP/IP stacks and subchannels” on page 100.) Concurrent LIC update The OSA features have increased memory to facilitate concurrent application of LIC updates, allowing the application of LIC updates without requiring a configuration off/on, thereby minimizing the disruption of networking traffic during the update. Concurrent LIC update applies to the OSA-Express3 and OSA-Express2 features (1000BASE-T Ethernet, Gigabit Ethernet SX, Gigabit Ethernet LX, 10 Gigabit Ethernet SR, and 10 Gigabit Ethernet LR). It is offered for the QDIO and OSA for NCP mode only (CHPID type OSD, OSM, OSX, and OSN). 98 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Hardware assists Complementary virtualization technology is available, which includes: 򐂰 QDIO Buffer-State Management (QEBSM): Two hardware instructions help to eliminate the overhead of hypervisor interception. 򐂰 Host Page-Management Assist (HPMA): An interface to the z/VM main storage management function designed to allow the hardware to assign, lock, and unlock page frames without z/VM hypervisor assistance. These hardware assists allow a cooperating guest operating system to initiate QDIO operations directly to the applicable channel, without interception by z/VM, thereby helping to provide additional performance improvements. Support is integrated in System z Licensed Internal Code. QDIO Diagnostic Synchronization for z/OS QDIO Diagnostic Synchronization is exclusive to System z, and the OSA-Express3 and OSA-Express2 features when configured as CHPID type OSD (QDIO). It is designed to provide the system programmer and network administrator with the ability to coordinate and simultaneously capture both operating system (software) and OSA (hardware) traces at the same instance of a system event. This allows the host operating system to signal the OSA-Express3 and OSA-Express2 feature to stop traces and capture the current trace records. Using existing tools (traps) and commands, the operator can capture both hardware and software traces at the same time, and then correlate the records during post processing. OSA-Express Network Traffic Analyzer for z/OS OSA-Express Network Traffic Analyzer is exclusive to System z and the OSA-Express3 and OSA-Express2 features when configured as CHPID type OSD (QDIO). It allows trace records to be sent to the host operating system to improve the capability to capture data for both the system programmer and the network administrator. This function allows the operating system to control the sniffer trace for the LAN and capture the records into host memory and storage, using existing host operating system tools to format, edit, and process the sniffer records. 7.1.3 Non-QDIO mode Like any other channel-attached control unit and device, an OSA port can execute channel programs (CCW chains) and present I/O interrupts to the issuing applications. For non-QDIO mode, the OSA ports are defined as channel type OSE. The non-QDIO mode requires the use of the OSA/SF for setup and customization of the OSA features. The 1000BASE-T features support non-QDIO mode. This mode supports SNA/APPN/HPR and TCP/IP traffic simultaneously through the OSA port. The non-QDIO mode types are as follows: TCP/IP Passthru In TCP/IP Passthru mode, an OSA feature transfers data between a TCP/IP stack to which it is defined and clients on an Ethernet 10/100/1000 Mbps LAN that is attached to the port on a 1000BASE-T feature and supports one of the following frame protocols: 򐂰 Ethernet II using the DEC Ethernet V 2.0 envelope 򐂰 Ethernet 802.3 using the 802.2 envelope with SNAP For TCP/IP Passthru mode, the default OAT may be used. In that case, no configuration or setup is required. Chapter 7. Open Systems Adapter-Express 99 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm SNA/APPN/HPR support In this mode, an OSA feature acts as an SNA Passthru agent to clients that use the SNA protocol on the LAN that is directly attached to the OSA-Express. If an OSA feature is running in the SNA mode, it is viewed by VTAM as an external communications adapter (XCA) that can have either switched or non-switched lines of communication. 7.1.4 OSA addressing support This section describes the maximum number IP addresses, MAC addresses, and subchannels supported by the OSA features. Maximum IP addresses per OAT The OSA Address Table (OAT) is a component of an OSA feature’s configuration. An OAT entry defines the data path between an OSA feature port and a logical partition (LP) and device unit address. That is, it manages traffic through the OSA CHPID. OSA-Express features support a maximum of 2048 IP addresses (IPv4, IPv6, and VIPA) per port, while OSA-Express3 and OSA-Express2 features support up to 4096 IP addresses per port. When the OSA port is defined in QDIO mode, the OAT table entries are built and updated dynamically. Maximum number of media access control (MAC) addresses When configured as CHPID type OSD, up to 2048 MAC or virtual (VMAC) addresses are supported per port with OSA features. Included in the maximum number of MAC addresses is the “burnt-in” MAC address of the OSA port. The MAC or VMAC addresses are added to the Layer 2 table of the OAT when the TCP/IP stacks (in which the addresses are defined) are started. Also see 7.2.13, “Layer 2 support” on page 110, and 7.2.16, “Layer 3 VMAC for z/OS” on page 113. Maximum TCP/IP stacks and subchannels A subchannel is a logical representation of a device. One subchannel is assigned to each device defined to the logical partition. Therefore, if you are sharing an OSA CHPID across 15 LPs and define one device, that device uses 15 subchannels. The maximum number of supported TCP/IP stacks and subchannels on z196, z10, and z9 servers are as follows: 򐂰 OSA port in non-QDIO mode (CHPID type OSE) An OSA port in non-QDIO mode is capable of supporting up to 120 TCP/IP stacks and 240 subchannels for all System z servers. 򐂰 OSA port in QDIO mode (CHPID type OSD) The OSA features support 640 TCP/IP stack connections per dedicated CHPID, or 640 total stacks across multiple logical partitions using a shared or spanned CHPID. The maximum number of subchannels allowed is 1920 (1920 subchannels / 3 = 640 stacks). 100 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Note: By default, OSA-Express3 and OSA-Express2 have multiple priorities for outbound queues enabled (four QDIO priorities). This means the maximum number of supported subchannels is reduced to 480 (1920 subchannels / 4 = 480 subchannels), thus reducing the total number of supported TCP/IP stacks to 160 (480 subchannels / 3 = 160 stacks). Priority queues can be disabled via HCD/IOCP. For example, in IOCP use the CHPARM=02 value to disable priority queuing. 7.1.5 OSA/SF support OSA/SF includes a Java™-based Graphical User Interface (GUI) in support of the client application. The Java GUI is independent of any operating system or server (transparent to the operating system), and is expected to operate wherever the current Java runtimes are available. Use of the GUI is optional. A REXX command interface is also included with OSA/SF. OSA/SF is not required to set up the OSA features in QDIO mode (CHPID type OSD), but it can be used for monitoring and controlling ports. OSA/SF has been, and continues to be, integrated in z/OS, z/VM, and z/VSE, and runs as a host application. For OSA/SF, Java GUI communication is supported via TCP/IP only. In the past, communication was supported via EHLLAPI (3270), APPC, and TCP/IP. This integrated version of OSA/SF is a complete replacement for the currently integrated versions in z/OS, z/VM, and z/VSE. This version of OSA/SF is not being offered as a separately orderable program product. The Open Systems Adapter Support Facility (OSA/SF) is used primarily to: 򐂰 Manage all OSA ports. 򐂰 Configure all OSA non-QDIO ports. 򐂰 Configure local MAC. 򐂰 Display registered IPv4 addresses (in use and not in use). It is supported on System z servers for QDIO ports. 򐂰 Display registered IPv4 or IPv6 Virtual MAC and VLAN ID associated with all OSA Ethernet features configured as QDIO Layer 2. 򐂰 Provide status information about an OSA port. Its shared or exclusive use state. OSA/SF is an integrated component of z/VM. This support is applicable to all OSA features on System z servers. With z/OS, a second interface using a set of REXX EXECs through the Time Sharing Option Extensions (TSO/E) can be used to control the OSA features defined to System z servers on which the TSO/E is running. OSA/SF is not always required to customize an OSA feature, but is highly recommended to gather operational information and to assist in problem determination. The OSA/SF Query function provides performance information about the OSA CHPIDs. OSA/SF is not required to configure the OSA features in operating modes OSD, OSC, and OSN. Chapter 7. Open Systems Adapter-Express 101 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2 OSA capabilities This section discusses the capabilities that exploit the OSA-Express3 and OSA-Express2 features. 7.2.1 Virtual IP address (VIPA) In the TCP/IP environment, VIPA frees TCP/IP hosts from dependence on a particular network attachment, allowing the establishment of primary and secondary paths through the network. VIPA is supported by all of the OSA features. An IP address traditionally ties to a physical link at one end of a connection. If the associated physical link goes down, it will be unreachable. The Virtual IP Address, on the other hand, exists only in software and has no association to any physical link. The TCP/IP stack is the destination IP address instead of the network attachment. VIPA provides for multiple IP addresses to be defined to a TCP/IP stack, allowing fault-tolerant, redundant, backup paths to be established. Applications become insensitive to the condition of the network since the VIPA will always be active, enabling users to route around intermediate points of failure in the network. VIPA Takeover and Takeback Since a VIPA is associated with a TCP/IP stack and not a physical network attachment, it can be moved to any TCP/IP stack within its network. If the TCP/IP stack that the VIPA is on fails (due to an outage), the same VIPA can be brought up automatically on another TCP/IP stack (VIPA Takeover) to allow end users to reach the backup server and applications. The original session between the user and original server is not disrupted. Once the failed TCP/IP stack is restored, the same VIPA can be moved back automatically (VIPA Takeback). 7.2.2 Primary/secondary router function The primary/secondary router function enables an OSA port to forward packets with unknown IP addresses to a TCP/IP stack for routing through another IP network interface, such as HiperSockets or another OSA feature. In order for an OSA port to forward IP packets to a particular TCP/IP stack for routing to its destination, the PRIRouter must be defined on the DEVICE statement in the TCP/IP profile. If the TCP/IP stack that has an OSA port defined as PRIRouter becomes unavailable, then a second TCP/IP stack, defined as the secondary router (SECRouter on the DEVICE statement in the TCP/IP profile), will receive the packets for unknown IP addresses. For enhanced availability, the definition of one primary router and multiple secondary routers for devices on an OSD-type CHPID is supported; however, only one secondary router is supported for devices on an OSE-type CHPID. Important: Sharing a single OSA port can fail in Load Balancing solutions. A circumvention is to use GRE or NAT, which can have a negative effect on performance. Layer 3 virtual MAC is a function available on System z servers with OSA-Express. For more detailed information about Layer 3 VMAC for z/OS refer to 7.2.16, “Layer 3 VMAC for z/OS” on page 113. 102 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.3 IPv6 support Internet Protocol Version 6 (IPv6) is supported by the OSA features when configured in QDIO mode. IPv6 is the protocol designed by the Internet Engineering Task Force (IETF) to replace Internet Protocol Version 4 (IPv4). IPv6 provides improved traffic management in the following areas: 򐂰 128-bit addressing Eliminates all practical limitations on global address ability. This means that private address space—and the network address translators (NATs) used between private intranet and public Internet—are no longer needed. 򐂰 Simplified header formats Allow for more efficient packet handling and reduced bandwidth cost. 򐂰 Hierarchical addressing and routing Keep routing tables small and backbone routing efficient by using address prefixes rather than address classes. 򐂰 Improved support for options Changes the way IP header options are encoded, allowing more efficient forwarding and greater flexibility. 򐂰 Address auto-configuration Allows stateless IP address configuration without a configuration server. In addition, IPv6 brings greater authentication and privacy capabilities through the definition of extensions, and integrated quality of service (QoS) through a traffic class byte in the header. Chapter 7. Open Systems Adapter-Express 103 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.4 Large send for TCP/IP traffic Large send can improve performance by offloading TCP packet processing from the host to the OSA-Express3 or OSA-Express2 features running in QDIO mode. Offload allows the host to send large blocks of data (up to 64 kilobytes) directly to the OSA-Express3 or OSA-Express2 feature. The OSA-Express3 or OSA-Express2 feature then fragments those large blocks into standard Ethernet frames (1500 bytes) to be sent out on the LAN (Figure 7-3). Application send buffer Ethernet frame Jumbo frame TCP large send 63 KB 63 KB 63 KB 1.5 KB TCP Stack OSA-Express2 or OSA-Express3 9 KB .. .. . 9 KB 1.5 KB 9 KB 1.5 KB .. .. . 63 KB .. 1.5 KB 1.5 KB .. .. . 1.5 KB .. Figure 7-3 Large send versus standard Ethernet and Jumbo frame sizes Large send supports outbound IPv4 traffic only, and applies solely to unicasts. Large send support reduces host processor utilization, returning CPU cycles for application use while increasing network efficiencies. Large send applies only to the OSA-Express3 and OSA-Express2 features. 7.2.5 VLAN support Virtual Local Area Network (VLAN) is supported by the OSA-Express3 and OSA-Express2 features when configured in QDIO mode. This support is applicable to z/OS, z/VM, and Linux on System z environments. The IEEE standard 802.1q describes the operation of Virtual Bridged Local Area Networks. A VLAN is defined to be a subset of the active topology of a Local Area Network. The OSA features provide for the setting of multiple unique VLAN IDs per QDIO data device. They also provide for both tagged and untagged frames to flow from an OSA port. The number of VLANs supported is specific to the operating system. VLANs facilitate easy administration of logical groups of stations that can communicate as though they were on the same LAN. They also facilitate easier administration of moves, adds, and changes in members of these groups. VLANs are also designed to provide a degree of low-level security by restricting direct contact with a server to only the set of stations that comprise the VLAN. 104 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm With System z servers, where multiple TCP/IP stacks exist, potentially sharing one or more OSA features, VLAN support is designed to provide a greater degree of isolation (Figure 7-4). LP 1 z/OS TCP/IP IPv4 VLAN 28 LP 2 z/OS TCP/IP IPv4 IPv6 VLAN16 VLAN37 Port 1 LP 3 Linux TCP/IP IPv6 IPv4 VLAN37 VLAN4 Port 2 LP 4 z/VM TCP/IP IPv4 VLAN12 OSA-Express Ethernet ports in QDIO mode Ethernet Switch VLAN28 VLAN4 VLAN16 VLAN12 Common physical network Figure 7-4 VLAN support VLAN support for z/OS Full Virtual Local Area Network (VLAN) support is offered for all OSA Ethernet features available on System z servers. z/OS Communications Server supports Virtual Local Area Network Identifications (VLAN IDs). Support is offered for up to eight global VLAN IDs per OSA port, based on the IP version: 򐂰 Eight Global VLAN (IDs) for IPv4 򐂰 Eight Global VLAN (IDs) for IPv6 VLAN support for z/VM z/VM exploits the VLAN technology and conforms to the IEEE 802.1q standard. Support is offered for one global VLAN ID per OSA port, based on the IP version: 򐂰 One Global VLAN (ID) for IPv4 򐂰 One Global VLAN (ID) for IPv6 The z/VM TCP/IP stack supports one VLAN ID per OSA port. Each port can be configured with a different VLAN ID. VLAN support for Linux on System z VLAN support in a Linux on System z environment is available for the OSA Ethernet features operating in QDIO mode. For Linux on System z support, refer to the following Web site for further information: http://www.ibm.com/developerworks VLAN support of GVRP GVRP is defined in the IEEE 802.1p standard for the control of IEEE 802.1q VLANs. It can be used to help simplify networking administration and management of VLANs. Chapter 7. Open Systems Adapter-Express 105 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm With GVRP support, an OSA-Express3 or OSA-Express2 port can register or de-register its VLAN IDs with a GVRP-capable switch and dynamically update its table as the VLANs change (Figure 7-5). OSA-Express2 OSA-Express3 VLAN22 VLAN33 VLAN44 VLAN 22 VLAN33 GVRP support dynamically registers VLAN IDs to the physical LAN VLAN44 Physical LAN Figure 7-5 GVRP support Support of GVRP is exclusive to System z. It is applicable to all of the OSA-Express3 and OSA-Express2 features when in QDIO mode (CHPID type OSD), and is supported by z/OS and z/VM. 7.2.6 SNMP support for z/OS and Linux on System z SNMP is supported for all of the OSA features when configured in the QDIO mode (CHPID type OSD). The OSA SNMP subagent support offered via the OSA features LIC includes: 򐂰 Get and GetNext requests This support applies to all OSA features supported on System z servers. 򐂰 dot3StatsTable Ethernet data for dot3StatsTable applies to all of the Ethernet features supported on System z servers. It implements the SNMP EtherLike Management Information Base (MIB) module in RFC 2665, which provides statistics for Ethernet interfaces. These statistics can assist in the analysis of network traffic congestion. 򐂰 Performance data This support applies to all of the OSA features supported on System z servers. The performance data reflects the OSA utilization. 򐂰 Traps and Set This support applies to all of the OSA features supported on System z. SNMP support for LAN Channel Station (LCS) applies to all of the OSA features supported on System z, in conjunction with TCP/IP applications only. It supports the same SNMP requests and alerts offered in QDIO mode (Get, GetNext, Trap, and Set), and is exclusive to the z/OS environment. For more information about SNMP support refer to: http://www.ibm.com/servers/eserver/zseries/networking/dsnmp.html 106 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Tip: If you subscribe to the document “OSA-Express Direct SNMP MIB module” through Resource Link, you will receive e-mail notification of document changes. Open Systems Adapter Support Facility is not required to manage SNMP data for the OSA features. An SNMP subagent exists on an OSA feature, which is part of a direct path between the z/OS or Linux on System z master agent (TCP/IP stacks) and an OSA-Express Management Information Base (MIB). The OSA features support an SNMP agent by providing data for use by an SNMP management application, such as Tivoli NetView®. This data is organized into MIB tables defined in the TCP/IP enterprise-specific MIB, as well as standard RFCs. The data is supported by the SNMP TCP/IP subagent (Figure 7-6). Agents (servers) Managers (clients) z/OS z/OS UNIX Shell User's Address Space osnmp command Tivoli NetView Address Space SNMP command OSAD device with subagent OSNMPD SNMP Agent's Address Space TCP/IP Address Space TCP SNMP Subagent OSA proxy Subagent System z OSA-Express Figure 7-6 SNMP support - z/OS example 7.2.7 TCP/IP multicast and broadcast support Multicast and broadcast support is part of the Internet Protocol assist (IPA) function of the OSA feature. Multicast support For sending data to multiple recipients, OSA features support IP multicast destinations only in QDIO or IP Passthru mode. TCP/IP broadcast support for z/OS, z/VM, and Linux on System z Broadcast support is included for all of the OSA features when configured in QDIO mode and supporting the Routing Information Protocol (RIP) Version 1. Broadcast is also supported for all of the OSA features when carrying TCP/IP traffic and configured in the non-QDIO mode (LAN Channel Station - LCS mode). A broadcast simultaneously transmits data to more than one destination; messages are transmitted to all stations in a network (for example, a warning message from a system operator). The broadcast frames can be propagated through an OSA feature to all TCP/IP applications that require broadcast support, including applications using RIP V1. Chapter 7. Open Systems Adapter-Express 107 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.8 ARP cache management The query and purge ARP enhancements are supported for all OSA features when configured in QDIO mode. The OSA feature maintains a cache of recently acquired IP-to-physical address mappings (or bindings). When the binding is not found in the ARP cache, a broadcast (an ARP request “How can I reach you?”) to find an address mapping is sent to all hosts on the same physical network. Because a cache is maintained, ARP does not have to be used repeatedly, and the OSA feature does not have to keep a permanent record of bindings. Query ARP table for IPv4 for Linux on System z The Query ARP table is supported using Internet Protocol Version 4 (IPv4). The TCP/IP stack already has an awareness of Internet Protocol Version 6 (IPv6) addresses. Purge ARP entries in cache for IPv4 for z/OS and Linux on System z Purging of entries in the ARP cache is supported using IPv4. The TCP/IP stack already has an awareness of IPv6 addresses. ARP takeover ARP takeover provides the capability of switching OSA port operations from one OSA to another OSA running in the same mode in z/OS environments. When z/OS TCP/IP is started in QDIO mode, it downloads all the home IP addresses in the stack and stores them in each OSA feature to which it has a connection. This is a service of QDIO architecture and only occurs automatically for OSD channels. For OSA ports set up as OSE channels (non-QDIO), you must define multiple IP addresses in the OSA Address Table using OSA/SF. The OSA then responds to ARP requests for its own IP address, as well as for virtual IP addresses (VIPAs). If an OSA feature fails while there is a backup OSA available on the same network or subnetwork, TCP/IP informs the backup OSA which IP addresses (real and VIPA) to take over, and the network connection is maintained. Note that for this to work, multiple paths must be defined to the TCP/IP stack. For example, MULTIPATH must be defined to the IPCONFIG statement of the TCP/IP profile in z/OS. ARP statistics QDIO includes an IP assist (IPA) function, which gathers Address Resolution Protocol (ARP) data during the mapping of IP addresses to media access control (MAC) addresses. CHPIDs defined as OSD maintain ARP cache information in the OSA feature (ARP offload). This is useful in problem determination for the OSA feature. Note that not all OSA features provide ARP counter statistics and ARP cache information to TCP/IP. 7.2.9 IP network availability There are several ways to ensure network availability, should failure occur at either the logical partition or the CHPID/network connection level. Port sharing, redundant paths, and the use of primary and secondary ports all provide some measure of recovery. A combination of these can guarantee network availability regardless of the failing component. When TCP/IP is started in QDIO mode, it downloads all the home IP addresses in the stack and stores them in the OSA feature. This is a service of QDIO architecture. The OSA port then responds to ARP requests for its own IP address, as well as for virtual IP addresses (VIPAs). If an OSA feature fails while there is a backup OSA available on the same network or subnetwork, TCP/IP informs the backup OSA port which IP addresses (real and VIPA) to take 108 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm over, and sends a gratuitous ARP that contains the MAC address of the backup OSA-Express. The network connection is maintained. 7.2.10 Checksum offload support for z/OS and Linux on System z z/OS and Linux on System z environments provide the capability of calculating and validating the Transmission Control Protocol/User Datagram Protocol (TCP/UDP) and Internet Protocol (IP) header checksums. Checksums are used to verify the contents of files when transmitted over a network. For example: 򐂰 OSA will validate the TCP, UDP, and IP header checksums for inbound packets. 򐂰 OSA will calculate the TCP, UDP, and IP header checksums for outbound packets. Checksum offload is supported by all OSA Ethernet features when operating in QDIO mode. By offloading checksum processing to the supporting OSA features, host server cycles are reduced, which can result in improved performance for most IPv4 packets. Note: Linux on System z only supports inbound checksum offload (inbound packets). When checksum is offloaded, the OSA feature performs the checksum calculations. Therefore, this function only applies to packets which actually go onto the LAN or come in from the LAN. When multiple IP stacks share an OSA port, and an IP stack sends a packet to a next hop IP address owned by another IP stack sharing the same OSA port, OSA sends the IP packet directly to the other IP stack without placing it out on the LAN. Checksum offload does not apply to such IP packets. Checksum offload does not apply to IPv6 packets. TCP/IP will continue to perform all checksum processing for IPv6 packets. This function does not apply to ICMP checksum processing. TCP/IP will continue to perform processing for ICMP checksum. Checksum offload support is available with z/OS. For Linux on System z support, refer to the following Web site for further information: http://www.ibm.com/developerworks 7.2.11 Dynamic LAN idle for z/OS Dynamic LAN idle is exclusive to System z servers and applies to the OSA-Express3 and OSA-Express2 features (CHPID type OSD), and is supported by z/OS. Dynamic LAN idle is designed to reduce latency and improve networking performance by dynamically adjusting the inbound blocking algorithm. When enabled, the z/OS TCP/IP Stack will adjust the inbound blocking algorithm to best match the application requirements. For latency sensitive applications, the blocking algorithm is modified to be latency sensitive. For streaming (throughput sensitive) applications, the blocking algorithm is adjusted to maximize throughput. In all cases, the z/OS TCP/IP stack dynamically detects the application requirements, making the necessary adjustments to the blocking algorithm. The monitoring of the application and the blocking algorithm adjustments are made in real-time, dynamically adjusting the application’s LAN performance. System administrators can authorize the z/OS TCP/IP stack to enable a dynamic setting, which was previously a static setting. The z/OS TCP/IP stack dynamically determines the best setting for the current running application, based on system configuration, system, inbound workload volume, CPU utilization, traffic patterns, and other related items. Chapter 7. Open Systems Adapter-Express 109 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.12 QDIO optimized latency mode QDIO optimized latency mode (OLM) can help improve performance for applications that have a critical requirement to minimize response times for inbound and outbound data. OLM optimizes the interrupt processing as follows: 򐂰 For inbound processing, the TCP/IP stack looks more frequently for available data to process, ensuring that any new data is read from the OSA-Express3 without requiring additional program controlled interrupts (PCIs). 򐂰 For outbound processing, the OSA-Express3 also looks more frequently for available data to process from the TCP/IP stack, thus not requiring a Signal Adapter (SIGA) instruction to determine whether more data is available. OLM is supported by the Communications Server for z/OS V1R11 with any OSA-Express3 feature on z196 and z10. See the 2817DEVICE, 2097DEVICE and the 2098DEVICE Preventive Service Planning (PSP) buckets for further information. 7.2.13 Layer 2 support The OSA Ethernet features on System z servers can support two transport modes of the OSI model: Layer 2 (Link Layer or MAC Layer) and Layer 3 (Network Layer). The Layer 2 transport mode allows for communication with IP and non-IP protocols. OSA works in conjunction with either z/VM TCP/IP or Linux on System z Layer 2 support running in a Logical Partition or as a z/VM guest. The z/VM virtual switch can also be used to enable Layer 2 functionality for guest systems; this is illustrated in Figure 7-7. z/VM Linux guest z/VM guest 02-00-00-00-00-01 02-00-00-00-00-02 TCP/UDP Appl. Linux SNA Appl. Virtual Switch (Layer 2) 02-00-00-00-00-03 Ethernet Switch NetBIOS Appl. 02-00-00-00-00-01 02-00-00-00-00-02 02-00-00-00-00-03 OSA-Express Figure 7-7 Layer 2 support for OSA-Express The virtual switch exploits both Layer 2 and Layer 3 support in the z/VM Control Program. For Layer 2 support, the z/VM Control Program owns the connection to the OSA feature and manages the MAC addresses and VLAN connectivity of the attached guest systems. The virtual switch performs automatic MAC address generation and assignment to allow 110 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm uniqueness across the z/VM guest systems. MAC addresses can also be locally administered. The virtual switch uses each guest system’s unique MAC address to forward frames. Data is transported and delivered within Ethernet frames, providing the ability to transport both IP and non-IP (for example, NetBIOS and SNA) frames through the fabric that the virtual switch supports. Through the address-resolution process each guest system’s MAC address becomes known to hosts residing on the physical side of the LAN segment. All inbound or outbound frames passing through the OSA port have the guest system’s corresponding MAC address as the source or destination address. The OSA Ethernet features can filter inbound frames by Virtual Local Area Network identification (VLAN ID, IEEE 802.1q), the Ethernet destination MAC address, or both. Filtering can reduce the amount of inbound traffic being processed by the operating system, helping to reduce CPU utilization. Filtering by VLAN ID or MAC address can also allow you to isolate portions of your environment that have sensitive data, thereby providing a degree of low-level security. Link aggregation for z/VM in Layer 2 mode Link aggregation is exclusive to System z and is applicable to the OSA-Express2 and OSA-Express3 features in Layer 2 mode when configured as CHPID type OSD (QDIO), and is supported by z/VM. z/VM virtual switch-controlled (VSWITCH-controlled) link aggregation (IEEE3 802.3ad) allows you to dedicate an OSA-Express3 or OSA-Express2 port to the z/VM operating system when the port is participating in an aggregated group when configured in Layer 2 mode. Link aggregation (trunking) is designed to allow you to combine multiple physical OSA-Express3 or OSA-Express2 ports into a single logical link for increased throughput and for nondisruptive failover in the event that a port becomes unavailable. Link aggregation for z/VM in Layer 2 mode functions as follows: 򐂰 Aggregated links are viewed as one logical trunk and contain all of the Virtual LANs (VLANs) required by the LAN segment. 򐂰 Link aggregation is between a VSWITCH and the physical network switch. 򐂰 Load balance communications is across multiple links in a trunk to prevent a single link from being overrun. 򐂰 Up to eight OSA-Express3 or OSA-Express2 ports are supported in one aggregated link. 򐂰 Ability to dynamically add/remove OSA ports for on demand bandwidth. 򐂰 Full-duplex mode (send and receive). Note: Target links for aggregation must be of the same type (for example, Gigabit Ethernet to Gigabit Ethernet). 7.2.14 QDIO data connection isolation for z/VM The QDIO data connection isolation function provides a higher level of security on z196, z10, and z9 servers when sharing the same OSA port in z/VM environments that use the virtual switch (VSWITCH). The VSWITCH is a virtual network device that provides switching between OSA ports and the connected guest systems. QDIO data connection isolation (also known as VSWITCH port isolation), allows you to disable the internal routing on a per QDIO connection basis, providing a means for creating Chapter 7. Open Systems Adapter-Express 111 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm security zones and preventing network traffic between the zones. For example, this function will ensure that traffic to an external network is forced to flow only between a guest operating system and the external network (Figure 7-8). z/VM LPAR Linux Guest z/OS Guest Virtual Switch port isolation Linux Guest z/OS Guest OSA-Express3 Ethernet Switch Figure 7-8 VSWITCH port isolation As shown in Figure 7-8, when in isolation mode, data traffic destined for a guest system port in the VSWITCH will be blocked, while traffic going to an external host will be sent to the OSA port for delivery. The isolation options (ON or OFF) can be set via the SET VSWITCH ISOLATION command. QDIO data connection isolation is supported by all OSA-Express3 and OSA-Express2 features on z196 and z10, and by all OSA-Express2 features on z9, with an MCL update. Refer to the appropriate Preventive Service Planning bucket for details regarding your System z server. 7.2.15 QDIO interface isolation for z/OS Some environments require strict controls for routing data traffic between severs or nodes. In certain cases, the LPAR-to-LPAR capability of a shared OSA port can prevent such controls from being enforced. With interface isolation, internal routing can be controlled on an LPAR basis. When interface isolation is enabled, the OSA will discard any packets destined for a z/OS LPAR that is registered in the OAT as isolated. 112 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm For example, in Figure 7-9, LPAR 1 has interface isolation enabled. Therefore, data traffic from LPAR 2 destined for LPAR 1 will be dropped by the OSA. System z10 server z/OS z/VM z/OS LPAR 1 LPAR 2 LPAR 3 OSA-Express3 Ethernet Switch Figure 7-9 QDIO interface isolation QDIO interface isolation is supported by the Communications Server for z/OS V1R11 and all OSA-Express3 and OSA-Express2 features on z196 and z10, and by all OSA-Express2 features on z9, with an MCL update. Refer to the appropriate Preventive Service Planning bucket for details regarding your System z server. 7.2.16 Layer 3 VMAC for z/OS To help simplify the infrastructure and provide load balancing when a logical partition is sharing the same OSA MAC address with another logical partition, each operating system instance can now have its own unique virtual MAC (VMAC) address. All IP addresses associated with a TCP/IP stack are accessible using their own VMAC address, instead of sharing the MAC address of an OSA port. This applies to Layer 3 mode and to an OSA port shared among logical partitions. This support is designed to: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Improve IP workload balancing Dedicate a Layer 3 VMAC to a single TCP/IP stack Remove the dependency on Generic Routing Encapsulation (GRE) tunnels Improve outbound routing Simplify configuration setup Allow z/OS to use a standard interface ID for IPv6 addresses Allow WebSphere® Application Server content-based routing to support an IPv6 network Eliminate the need for PRIROUTER/SECROUTER function in z/OS OSA Layer 3 VMAC for z/OS is applicable to OSA Ethernet features when configured as CHPID type OSD (QDIO). Chapter 7. Open Systems Adapter-Express 113 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.17 Enterprise Extender The Enterprise Extender (EE) function of z/OS Communications Server allows you to run SNA applications and data on IP networks and IP-attached clients. It can be used with any OSA feature running IP traffic. EE is a simple set of extensions to the open High Performance Routing technology that integrate HPR frames into User Datagram Protocol/Internet Protocol (UDP/IP) packets, providing: 򐂰 SNA application connectivity using an IP backbone support for: – SNA-style priority – SNA Parallel Sysplex exploitation 򐂰 Improved throughput and response times 򐂰 Compatible support for TCP and UDP traffic on the IP portion of the application traffic path (SNA/HPR and UDP/IP traffic can coexist on an EE connection.) The EE function is a TCP/IP encapsulation technology that carries SNA traffic from an endpoint over an IP network (for example, via the OSA port to Communications Server) to another endpoint where it is de-encapsulated and presented to an SNA application. EE requires APPN/HPR at the endpoints. In order to enable EE, you must configure the TCP/IP stack with a virtual IP address and define an XCA major node. The XCA major node is used to define the PORT, GROUP, and LINE statements for the EE connections. Refer to the Enterprise Extender Implementation Guide, SG24-7359, for more information. 7.2.18 TN3270E Server The TN3270E Server is supported by z/OS. It allows desktop users to connect through an IP network directly to SNA applications. The following support is provided: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Secure access using SSL and Client Authentication via RACF® Over 64,000 sessions per server when using multiple ports Over 2,000 transactions/second with sub-second response time Reconnect 16 K sessions in less than a minute using VIPA Takeover support IP QoS using Service Policy Server Host Print Support Tivoli support provides IP visibility to VTAM Manage with your data center resources More robust than external TN3270 servers z/OS Communications Server also supports a secure, RACF-based single sign-on logic called Express Logon Facility (ELF). ELF works with IBM TN3270 clients to securely authenticate the client, acquire a passtoken, and then pass it on to the TN3270E server for replacement/submission to the application. For more information about the TN3270E server, based on your release, refer to Communications Server for z/OS TCP/IP Implementation Volume 2: Standard Applications: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query="Communications+Server"+AND+"Volume+2" 7.2.19 OSA for NCP support Open Systems Adapter for NCP (OSA for NCP) support is available on z196, z10, and z9 servers. It provides channel connectivity from the operating systems to the Communication Controller for Linux on System z (CCL) image, using OSA-Express3 and OSA-Express2 114 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Ethernet features with CHPID type OSN. Each operating system that currently supports Channel Data Link Control (CDLC) can utilize this connectivity option without changes to the operating system. OSN supports both SNA PU Type 5 and PU Type 2.1 channel connectivity. OSA for NCP can help eliminate the requirement for having an external medium (and all related hardware) for communications between an operating system and the CCL image in the same System z server. Traffic between the logical partitions (operating system and CCL) is no longer required to flow on the Local Area Network (LAN) or ESCON channel. Utilizing existing SNA support (multiple transmission groups), OSN support permits multiple connections between the same CCL image and the same operating system (such as z/OS or z/TPF), residing in the same System z9 server as the CCL image. OSN provides an efficient method of communication, and is designed to create a secure and seamless integration of the operating system and CCL. For example, CHPID type OSN: 򐂰 Appears to the operating systems as an ESCON channel connected to a 374x device type, which exploits existing CDLC protocols. 򐂰 Allows system administrators of the various operating systems to configure, manage, and operate their CCL NCPs as though they were running in an ESCON-attached 374x Communications Controller. 򐂰 Enables NCP channel-related functions such as loading and dumping of the NCP. 򐂰 Does not require external hardware (cables or switches). 򐂰 Allows multiple CCL images to communicate with multiple operating system images, supporting up to 180 connections (374x devices). 򐂰 Can span channel subsystems. OSN support is exclusive to System z servers, and is supported by z/OS, z/VM, z/VSE, z/TPF, and Linux on System z. Chapter 7. Open Systems Adapter-Express 115 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Figure 7-10 shows the traffic flow on a System z server between VTAM on a z/OS logical partition or z/TPF logical partition communicating with the CCL running as a guest under z/VM in a separate logical partition. Note that one Ethernet OSA port (CHPID type OSN) is used. On the z/OS or z/TPF side, it is defined as CDLC, while on the CCL side it is defined as QDIO (QETH). A second Ethernet OSA port (CHPID type OSD) is defined as QETH and is used to communicate with SNA devices on the LAN. Linux on System z NCP CCL OSD or OSE z/OS, z/TPF VTAM CDLC OSN Traffic between the LPs Traffic between CCL and LAN environment Figure 7-10 CDLC traffic flow, using OSA for NCP For more information about CCL, refer to IBM Communication Controller for Linux on System z V1.2.1 Implementation Guide, SG24-7223, and the following URL: http://www.ibm.com/software/network/ccl 7.2.20 Adapter interruptions for QDIO Linux on System z and z/VM work together to provide performance improvements by exploiting extensions to the Queued Direct I/O (QDIO) architecture. Adapter interruptions, first added to z/Architecture with HiperSockets, provide an efficient, high-performance technique for I/O interruptions to reduce path lengths and overhead in both the host operating system and the adapter (OSA-Express3 and OSA-Express2 when using type OSD CHPID). In extending the use of adapter interruptions to OSD (QDIO) channels, the programming overhead to process a traditional I/O interruption is reduced. This benefits OSA-Express TCP/IP support in Linux on System z, z/VM and z/VSE. Adapter interruptions apply to all of the OSA-Express3 and OSA-Express2 features on z196 and z10 when in QDIO mode (CHPID type OSD). 7.2.21 Inbound workload queueing (IWQ) Inbound Workload Queuing (IWQ) has been designed to help reduce overhead and latency for inbound z/OS network data traffic, as well as implement an efficient way for initiating parallel processing. This is all achieved by using an OSA-Express3 feature in QDIO mode with multiple input queues and by processing network data traffic based on workload types. The data from a specific workload type is placed in one of four input queues and a process is 116 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm created and scheduled to execute on one of multiple processors, independent from the other three queues. This greatly improves performance, because IWQ can exploit the SMP architecture of zEnterprise 196 and System z10 severs. Figure 7-11 illustrates the IWQ concept of using multiple inbound queues for different types of workloads (T1 through T4), compared to a single inbound queue. z/OS LPAR z/OS LPAR App 1 App 2 App 3 App 4 App 1 App 2 App 4 TCP/IP TCP/IP stack T1 T2 CPU CPU 1 1 Process + deliver of all traffic stack T3 T4 CPU CPU CPU 2 3 4 Device Device OSA-Express App 3 Single inbound queue Multiple inbound queues OSA-Express3 LAN LAN Figure 7-11 Single inbound queue versus multiple inbound queues A primary objective of IWQ is to provide improved performance for business critical interactive workloads by reducing contention created by other types of workloads. The types of z/OS workloads that are identified and assigned to unique input queues are: 򐂰 z/OS Sysplex Distributor traffic - network traffic which is associated with a distributed virtual internet protocol address (VIPA) is assigned to a unique input queue allowing the Sysplex Distributor traffic to be immediately distributed to the target host 򐂰 z/OS bulk data traffic - network traffic which is dynamically associated with a streaming (bulk data) TCP connection is assigned to a unique input queue allowing the bulk data processing to be assigned the appropriate resources and isolated from critical interactive workloads. IWQ is exclusive to OSA-Express3 (CHPID types OSD and OSX1), and the z/OS operating system. It is supported by Communications Server for z/OS V1R12. See the 2817DEVICE, 2097DEVICE, and the 2098DEVICE Preventive Service Planning (PSP) buckets for further information. 1 Only supported on the zEnterprise 196 Chapter 7. Open Systems Adapter-Express 117 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.2.22 Network management - query and display OSA configuration As additional complex functions have been added to OSA, the ability for the system administrator to display, monitor, and verify the specific current OSA configuration unique to each operating system has become more complex. OSA-Express3 has the capability for the operating system to directly query and display the current OSA configuration information (similar to OSA/SF). z/OS exploits this OSA capability with a TCP/IP operator command called Display OSAINFO. Display OSAINFO allows the operator to monitor and verify the current OSA configuration, which helps improve the overall management, serviceability, and usability of OSA-Express3. Display OSAINFO is exclusive to OSA-Express3 (CHPID types OSD, OSM, and OSX), the z/OS operating system, and z/VM for guest exploitation for z196 and z10 servers. 7.2.23 OSA-Express3 for ensemble connectivity The following three types of OSA-Express3 features are used to connect the z196 central processor complex (CPC) to its attached IBM zEnterprise BladeCenter Extension (zBX), and other ensemble nodes: 򐂰 OSA-Express3 10 Gigabit Ethernet (GbE) Long Range (LR), feature code 3370 򐂰 OSA-Express3 10 Gigabit Ethernet (GbE) Short Reach (SR), feature code 3371 򐂰 OSA-Express3 1000BASE-T Ethernet, feature code 3367 Intraensemble data network (IEDN) The IEDN is a private and secure 10 Gbps Ehernet network that connects all elements of an ensemble and is access-controlled using integrated virtual LAN (VLAN) provisioning. No customer managed switches or routers are required. The IEDN is managed by a primary HMC2. IEDN requires two OSA-Express3 10 GbE ports (one port from two OSA-Express3 10 GbE features), configured as CHPID type OSX. The connection is from the z196 to the IEDN top of rack (TOR) switches on zBX. Or with a stand-alone z196 node the OSA Express-3 10 GbE ports are connected to one another in the z196 with client provided 10 GbE loop back cables (either SR or LR, depending on the OSA feature). Intranode management network (INMN) The INMN is a private and physically isolated 1000BASE-T Ethernet internal management network, operating at 1 Gbps. It connect all resources (z196 and zBX components) of an ensemble node for management purposes. It is prewired, internally switched, configured, and managed with full redundancy for high availability. The INMN requires two ports (one port from two OSA-Express3 1000BASE-T features), configured as CHPID type OSM. The connection is via port J07 of the bulk power hubs (BPHs) in the z196. The INMN top of rack (TOR) switches on zBX also connect to the BPHs. For detailed information about an ensemble network, see IBM zEnterprise System Technical Guide, SG24-7833. 2 118 This HMC must be running with Version 2.11 or above with feature codes 0090, 0025, 0019 and optionally 0020. IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.3 Connectivity The transmission medium and cabling requirements for the OSA ports depend on the OSA feature, OSA port type, and LAN. Acquiring cables and other connectivity items is the customer’s responsibility. Note: DIX V2 and IEEE 802.3 frames can coexist on the same network; however, clients or servers not using the same protocol will not be able to communicate with each other. OSA devices The different types of OSA channels (CHPID types) require the following device types: 򐂰 򐂰 򐂰 򐂰 OSA devices for QDIO (OSD) and non-QDIO (OSE) CHPID types 3270-X and 3287 devices for the OSA-ICC (OSC) CHPID type 3745 and OSN devices, for the OSA for NCP (OSN) CHPID type OSA devices for zBX (OSX) and ensemble management (OSM) CHPID types OSA/SF requires one device (defined via HCD) to be associated with the OSA CHPID as device type OSAD (UNITADD=FE). OSA/SF uses this device to communicate with the OSA feature. The OSA-Express Network Traffic Analyzer for z/OS requires one or more data paths devices for the OSAENTA trace interface, depending on the configuration. Multiple Image Facility (MIF) Multiple Image Facility (MIF) enables OSA ports installed on System z servers to be shared among logical partitions. For more information refer to Chapter 2, “Channel Subsystem overview” on page 13. Spanned channels Spanning is the ability to configure channels to multiple Channel Subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the Channel Subsystem to which the logical partition is configured. OSA ports can be spanned across multiple CSSs on System z servers. See 2.1.2, “Multiple CSS concept” on page 14, for more details on spanned channels. 7.3.1 OSA features Table 7-2 lists the OSA-Express3, OSA-Express2 and OSA-Express features supported on System z servers. The OSA-Express3 LAN adapters provide increased throughput, deliver double port density per card, and have reduced latency and overhead in the network traffic compared to the OSA-Express2 adapters. The maximum number of OSA ports supported by each server can be found in Table 7-2. Details for each feature that is supported on z196, z10 and z9 servers are in the subsequent sections. Table 7-2 System z OSA features Feature name OSA-Express GbE LX Feature code Connector type Cable type Maximum unrepeated distancea Server 1364 LC Duplex SM 9 µm 5 km z9 EC, z9 BCb MCPc 550 m (500) Chapter 7. Open Systems Adapter-Express 119 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm OSA-Express GbE LX 2364 OSA-Express GbE SX 1365 OSA-Express GbE SX 2365 SC Duplex LC Duplex SC Duplex SM 9 µm 5 km MCPc 550 m (500) MM 62.5 µm 220 m (166) 275 m (200) MM 50 µm 550 m (500) MM 62.5 µm 220 m (166) 275 m (200) MM 50 µm 550 m (500) z9 EC, z9 BCb z9 EC, z9 BCb z9 EC, z9 BCb OSA-Express 1000BASE-T 1366 RJ 45 UTP Cat5 100 m z9 EC, z9 BCb OSA-Express3 GbE LX 3362 LC Duplex 9 µm 5 km MCPc 550 m (500) z196, z10 EC, z10 BC OSA-Express3 GbE SX 3363 MM 62.5 µm 220 m (166) 275 m (200) MM 50 µm 550 m (500) SM 9 µm 5 km MCPc 550 m (500) MM 62.5 µm 220 m (166) 275 m (200) MM 50 µm 550 m (500) OSA-Express2 GbE LX 3364 OSA-Express2 GbE SX 3365 LC Duplex LC Duplex LC Duplex z196, z10 EC, z10 BC z196b , z10 EC, z10 BC z9 EC, z9 BC z196b , z10 EC, z10 BC z9 EC, z9 BC OSA-Express2 1000BASE-T 3366 RJ 45 UTP Cat5 100 m z196b , z10 EC, z10 BC z9 EC, z9 BC OSA-Express3 1000BASE-T 3367 RJ 45 UTP Cat5 100 m z196, z10 EC, z10 BC OSA-Express2 10 GbE LR 3368 SC Duplex SM 9 µm 10 km z10 EC, z10 BCb, z9 EC, z9 BC OSA-Express3-2P 1000BASE-T 3369 RJ 45 UTP Cat5 100 m z10 BC OSA-Express3 10 GbE LR 3370 LC Duplex SM 9 µm 10 km z196, z10 EC, z10 BC OSA-Express3 10 GbE SR 3371 LC Duplex MM 62.5 µm 33 m (200) MM 50 µm 300 m (2000) 82 m (500) z196, z10 EC, z10 BC OSA-Express3-2P GbE SX 3373 MM 62.5 µm 220 m (166) 275 m (200) MM 50 µm 550 m (500) LC Duplex z10 BC a. Minimum fiber bandwidth in MHz/km for multimode fiber optic links is included in parentheses where applicable. b. Carried forward only. c. Mode Conditioning Patch (MCP) cables enable the 1 Gbps single mode features to connect to multimode fiber. 120 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm OSA-Express3 10 Gigabit Ethernet Long Reach (10 GbE LR) feature Feature code 3370 is exclusive to the z196 and z10 servers. It occupies one slot and has two ports that connect to a 10 Gbps Ethernet LAN via a 9 micron single mode fiber optic cable terminated with an LC Duplex connector and supports an unrepeated maximum distance of 10 km (6.2 miles). The LC Duplex small form factor connector is a deviation from the SC Duplex connector used for the OSA-Express2 10 GbE LR feature. The OSA-Express3 10 GbE LR feature does not support auto-negotiation to any other speed and runs in full duplex mode only. OSA-Express3 10 GbE LR supports 64B/66B encoding, whereas GbE supports 8B/10 encoding, making auto-negotiation to any other speed impossible. The two ports of the OSA-Express3 10 GbE LR feature have two CHPIDs assigned and can be defined as type OSD or OSX. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10GBASE-LR (standard transmission scheme) – IEEE 802.3ae – IEEE 802.1q – IEEE 802.3x - flow control – DIX Version 2 OSA-Express3 10 Gigabit Ethernet Short Reach (10 GbE SR) feature Feature code 3371 is exclusive to the z196 and z10 servers. It occupies one slot and has two ports that connect to a 10 Gbps Ethernet LAN via a 62.5 micron or 50 micron multimode fiber optic cable terminated with an LC Duplex connector. The maximum supported unrepeated distance is 33 meters (108 feet) on a 62.5 micron multimode (200 MHz) fiber optic cable, 82 meters (269 feet) on a 50 micron multimode (500 MHz) fiber optic cable, and 300 meters (984 feet) on a 50 micron multimode (2000 MHz) fiber optic cable. The OSA-Express3 10 GbE SR feature does not support auto-negotiation to any other speed and runs in full duplex mode only. OSA-Express3 10 GbE LR supports 64B/66B encoding, whereas GbE supports 8B/10 encoding, making auto-negotiation to any other speed impossible. The two ports of the OSA-Express3 10 GbE SR feature have two CHPIDs assigned and can be defined as type OSD or OSX. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10GBASE-SR (standard transmission scheme) – IEEE 802.3ae – IEEE 802.1q – IEEE 802.3x - flow control – DIX Version 2 OSA-Express3 Gigabit Ethernet long wavelength (GbE LX) feature Feature code 3362 is exclusive to the z196 and z10 servers. It occupies one slot and has four ports that connect to a 1 Gbps Ethernet LAN via a 9 micron single mode fiber optic cable terminated with an LC Duplex connector, supporting an unrepeated maximum distance of 5 km (3.1 miles). Multimode (62.5 or 50 micron) fiber optic cable can be used with this Chapter 7. Open Systems Adapter-Express 121 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm features. The use of these multimode cable types requires a mode conditioning patch (MCP) cable at each end of the fiber optic link (see Table B-1 on page 199). Use of the single mode to multimode MCP cables reduces the supported distance of the link to a maximum of 550 meters (1084 feet). The four ports of the OSA-Express3 GbE LX feature have two CHPIDs assigned, so two corresponding ports share one CHPID number. The use of both ports on a two-port CHPID requires support by the operating system. The OSA-Express3 GbE LX feature does not support auto-negotiation to any other speed and runs in full duplex mode only. Each OSA-Express3 GbE LX port can be defined as CHPID type OSD or OSN. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 1000BASE-LX (standard transmission scheme) – – – – – IEEE 802.3ac IEEE 802.1q IEEE 802.3x IEEE 802.3z DIX Version 2 OSA-Express3 Gigabit Ethernet short wavelength (GbE SX) feature Feature code 3363 is exclusive to the z196 and z10 servers. It occupies one slot and has four ports that connect to a 1 Gbps Ethernet LAN via 50 or 62.5 micron multimode fiber optic cable terminated with an LC Duplex connector over an unrepeated distance of 550 meters (for 50 µm fiber) or 220 meters (for 62.5 µm fiber). The four ports of the OSA-Express3 GbE SX feature have two CHPIDs assigned, so two corresponding ports share one CHPID number. The use of both ports on a two-port CHPID requires support by the operating system. The OSA-Express3 GbE SX feature does not support auto-negotiation to any other speed and runs in full duplex mode only. Each OSA-Express3 GbE SX port can be defined as CHPID type OSD or OSN. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 1000BASE-SX (standard transmission scheme) – IEEE 802.3ac – IEEE 802.1q – IEEE 802.3x – IEEE 802.3z – DIX Version 2 OSA-Express3-2P Gigabit Ethernet short wavelength (GbE SX) feature Feature code 3373 is exclusive to the z10 BC server and occupies one slot in the I/O drawer. It has two ports that connect to a 1 Gbps Ethernet LAN via 50 or 62.5 micron multimode fiber optic cable terminated with an LC Duplex connector over an unrepeated distance of 550 meters (for 50 µm fiber) or 220 meters (for 62.5 µm fiber). 122 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm The two ports of the OSA-Express3-2P GbE SX feature have one CHPIDs assigned, so both ports share one CHPID number. The use of both ports on a two-port CHPID requires support by the operating system. The OSA-Express3-2P GbE SX feature does not support auto-negotiation to any other speed and runs in full duplex mode only. Each OSA-Express3-2P GbE SX port can be defined as CHPID type OSD or OSN. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 1000BASE-SX (standard transmission scheme) – IEEE 802.3ac – IEEE 802.1q – IEEE 802.3x – IEEE 802.3z – DIX Version 2 OSA-Express3 1000BASE-T Ethernet feature Feature code 3367 is exclusive to the z196 and z10 servers. It occupies one slot and has four ports that connect to a 1000 Mbps (1 Gbps), 100 Mbps, or 10 Mbps Ethernet LAN. Each port has an RJ-45 receptacle for cabling to an Ethernet switch. The RJ-45 receptacle is required to be attached using EIA/TIA category 5 unshielded twisted pair (UTP) cable with a maximum length of 100 meters (328 feet). The four ports of the OSA-Express3 1000BASE-T feature have two CHPIDs assigned, so two corresponding ports share one CHPID number. The use of both ports on a two-port CHPID requires support by the operating system. When you defined as CHPID type OSM, only port 0 is usable. The OSA-Express3 1000BASE-T Ethernet feature supports auto-negotiation when attached to an Ethernet router or switch. If you allow the LAN speed and duplex mode to default to auto-negotiation, the OSA port and the attached router or switch auto-negotiate the LAN speed and duplex mode settings between them, and connect at the highest common performance speed and duplex mode of interoperation. If the attached Ethernet router or switch does not support auto-negotiation, the OSA port examines the signal it is receiving, and connects at the speed and duplex mode of the device at the other end of the cable. You can choose any one of the following settings for the OSA-Express3 1000BASE-T Ethernet feature port: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Auto-negotiate 10 Mbps half-duplex 10 Mbps full-duplex 100 Mbps half-duplex 100 Mbps full-duplex 1000 Mbps full-duplex If you are not using auto-negotiate, the OSA port will attempt to join the LAN at the specified speed and duplex mode. If this does not match the speed and duplex mode of the signal on the cable, the OSA port will not connect. LAN speed and duplex mode can be set explicitly, using OSA/SF or the OSA Advanced Facilities function of the hardware management console (HMC). The explicit settings will override the OSA feature port’s ability to auto-negotiate with its attached Ethernet switch. Chapter 7. Open Systems Adapter-Express 123 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Each OSA-Express3 1000BASE-T port can be defined as CHPID type OSD, OSE, OSC, OSN or OSM. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10BASE-T (standard transmission scheme) – – – – IEEE 802.2 IEEE 802.3 ISO/IEC 8802-3 DIX Version 2 򐂰 100BASE-TX (standard transmission scheme) IEEE 802.3u 򐂰 1000BASE-T (standard transmission scheme) – – – – – – IEEE 802.1p IEEE 802.1q IEEE 802.3ab IEEE 802.3ac IEEE 802.3u IEEE 802.3x OSA-Express3-2P 1000BASE-T Ethernet feature Feature code 3369 is exclusive to the System z10 BC server and occupies one slot in the I/O drawer. It has two ports that connect to a 1000 Mbps (1 Gbps), 100 Mbps, or 10 Mbps Ethernet LAN. Each port has an RJ-45 receptacle for cabling to an Ethernet switch. The RJ-45 receptacle is required to be attached using EIA/TIA category 5 unshielded twisted pair (UTP) cable with a maximum length of 100 meters (328 feet). The two ports of the OSA-Express3-2P 1000BASE-T feature have one CHPIDs assigned, so the two corresponding ports share one CHPID number. The use of both ports on a two-port CHPID requires support by the operating system. The OSA-Express3-2P 1000BASE-T feature supports auto-negotiation when attached to an Ethernet router or switch. If you allow the LAN speed and duplex mode to default to auto-negotiation, the OSA port and the attached router or switch auto-negotiate the LAN speed and duplex mode settings between them, and connect at the highest common performance speed and duplex mode of interoperation. If the attached Ethernet router or switch does not support auto-negotiation, the OSA port examines the signal it is receiving, and connects at the speed and duplex mode of the device at the other end of the cable. You can choose any one of the following settings for the OSA-Express3-2P 1000BASE-T feature port: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Auto-negotiate 10 Mbps half-duplex 10 Mbps full-duplex 100 Mbps half-duplex 100 Mbps full-duplex 1000 Mbps full-duplex If you are not using auto-negotiate, the OSA port will attempt to join the LAN at the specified speed and duplex mode. If this does not match the speed and duplex mode of the signal on the cable, the OSA port will not connect. 124 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm LAN speed and duplex mode can be set explicitly, using OSA/SF or the OSA Advanced Facilities function of the hardware management console (HMC). The explicit settings will override the OSA-Express3-2p 1000BASE-T feature port’s ability to auto-negotiate with its attached Ethernet switch. Each OSA-Express3-2P 1000BASE-T port can be defined as CHPID type OSD, OSE, OSC, or OSN. See 7.1.1, “Operating modes” on page 94 for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10BASE-T (standard transmission scheme) – – – – IEEE 802.2 IEEE 802.3 ISO/IEC 8802-3 DIX Version 2 򐂰 100BASE-TX (standard transmission scheme) IEEE 802.3u 򐂰 1000BASE-T (standard transmission scheme) – – – – – – IEEE 802.1p IEEE 802.1q IEEE 802.3ab IEEE 802.3ac IEEE 802.3u IEEE 802.3x OSA-Express2 10 Gigabit Ethernet Long Reach (10 GbE LR) feature Feature code 3368 occupies one slot in the z196, z10 or z9 I/O cages or I/O drawers. It has one port that connects to a 10 Gbps Ethernet LAN via a 9 micron single mode fiber optic cable terminated with an SC Duplex connector and supports an unrepeated maximum distance of 10 km (6.2 miles). The OSA-Express2 10 GbE LR feature does not support auto-negotiation to any other speed and runs in full duplex mode only. The OSA-Express2 10 GbE LR port must be defined as CHPID type OSD. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10GBASE-LR (standard transmission scheme) – IEEE 802.3ae – IEEE 802.1q – IEEE 802.3x - flow control – DIX Version 2 OSA-Express2 Gigabit Ethernet long wavelength (GbE LX) feature Feature code 3364 occupies one slot in the z196, z10, or z9. It has two independent ports that connect to a 1 Gbps Ethernet LAN via a 9 micron single mode fiber optic cable terminated with an LC Duplex connector, supporting an unrepeated maximum distance of 5 km (3.1 miles). Multimode (62.5 or 50 micron) fiber optic cable can be used with this Chapter 7. Open Systems Adapter-Express 125 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm features. The use of these multimode cable types requires a mode conditioning patch (MCP) cable at each end of the fiber optic link (see Table B-1 on page 199). Use of the single mode to multimode MCP cables reduces the supported distance of the link to a maximum of 550 meters (1084 feet). The OSA-Express2 GbE LX feature does not support auto-negotiation to any other speed and runs in full duplex mode only. The OSA-Express2 GbE LX ports can be defined as CHPID type OSD or OSN. See 7.1.1, “Operating modes” on page 94 for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 1000BASE-LX (standard transmission scheme) – IEEE 802.3ac – IEEE 802.1q – IEEE 802.3x – IEEE 802.3z – DIX Version 2 OSA-Express2 Gigabit Ethernet short wavelength (GbE SX) feature Feature code 3365 occupies one slot in the z196, z10, or z9. It has two independent ports that connect to a 1 Gbps Ethernet LAN via 50 or 62.5 micron multimode fiber optic cable terminated with an LC Duplex connector over an unrepeated distance of 550 meters (for 50 µm fiber) or 220 meters (for 62.5 µm fiber). The OSA-Express2 GbE SX feature does not support auto-negotiation to any other speed and runs in full duplex mode only. The OSA-Express2 GbE SX ports can be defined as CHPID type OSD or OSN. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 1000BASE-SX (standard transmission scheme) – IEEE 802.3ac – IEEE 802.1q – IEEE 802.3x – IEEE 802.3z – DIX Version 2 OSA-Express2 1000BASE-T Ethernet feature Feature code 3366 occupies one slot in the z196, z10, or z9. It has two independent ports that connect to a 1000 Mbps (1 Gbps), 100 Mbps, or 10 Mbps Ethernet LAN. Each port has an RJ-45 receptacle for cabling to an Ethernet switch. The RJ-45 receptacle is required to be attached using EIA/TIA category 5 unshielded twisted pair (UTP) cable with a maximum length of 100 meters (328 feet). The OSA-Express2 1000BASE-T Ethernet feature supports auto-negotiation when attached to an Ethernet router or switch. If you allow the LAN speed and duplex mode to default to auto-negotiation, the OSA port and the attached router or switch auto-negotiate the LAN speed and duplex mode settings between them, and connect at the highest common performance speed and duplex mode of interoperation. If the attached Ethernet router or switch does not support auto-negotiation, the OSA port examines the signal it is receiving, and connects at the speed and duplex mode of the device at the other end of the cable. 126 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm You can choose any one of the following settings for the OSA-Express2 1000BASE-T Ethernet feature port: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Auto-negotiate 10 Mbps half-duplex 10 Mbps full-duplex 100 Mbps half-duplex 100 Mbps full-duplex 1000 Mbps full-duplex If you are not using auto-negotiate, the OSA port will attempt to join the LAN at the specified speed and duplex mode. If this does not match the speed and duplex mode of the signal on the cable, the OSA port will not connect. LAN speed and duplex mode can be set explicitly, using OSA/SF or the OSA Advanced Facilities function of the hardware management console (HMC). The explicit settings will override the OSA-Express2 feature port’s ability to auto-negotiate with its attached Ethernet switch. The OSA-Express2 1000BASE-T ports can be defined as CHPID type OSD, OSE, OSC, or OSN. See 7.1.1, “Operating modes” on page 94, for details on modes of operation and supported traffic types. The following Ethernet standards are applicable for this feature: 򐂰 10BASE-T (standard transmission scheme) – – – – IEEE 802.2 IEEE 802.3 ISO/IEC 8802-3 DIX Version 2 򐂰 100BASE-TX (standard transmission scheme) IEEE 802.3u 򐂰 1000BASE-T (standard transmission scheme) – – – – – – IEEE 802.1p IEEE 802.1q IEEE 802.3ab IEEE 802.3ac IEEE 802.3u IEEE 802.3x Note: It is intended that the z196 is the last server to support OSA-Express2 channels, We recommend that customers review the usage of their installed OSA-Express2 channels and where possible migrate to OSA-Express3 channels. 7.3.2 OSA function support Table 7-3 lists the functions that are supported based on OSA feature. Chapter 7. Open Systems Adapter-Express 127 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm Table 7-3 OSA function support Function GbE 1000BASE-T GbE 1000BASE-T ARP takeover x x xa x xa ARP cache management x x xa x xa Checksum offload support (IPv4 only) x x xa xb xa SNMP support x x x x x x xa Internet Protocol Version 6 (IPv6) support x x xa Large send support x x xa n/a n/a Inbound workload queueing (IWQ)c xa xa xa n/a n/a Layer 2 support x x xa x xa Layer 3 VMAC for z/OS x x x x x 640 TCP/IP (with priority queues disabled) x x xa n/a n/a Concurrent LIC update xd x x n/a n/a Primary/secondary router function x x x x x Multicast and broadcast support x x x x x Virtual IP address (VIPA) support x x x x x VLAN (IEEE 802.1q) support x x xa x xa VLAN support of GVRP xa xa,d xa,d n/a n/a x xd Jumbo frame support (8992 byte frame size) x x xe QDIO Diagnostic Synchronization for z/OSa x x x n/a n/a Network Traffic Analyzer for z/OSa x x x n/a n/a x x x n/a n/a x x x n/a n/a QDIO data connection isolationa x x x n/a n/a QDIO interface isolationa x x x n/a n/a Query and display OSA configurationc xa xa xa n/a n/a Dynamic LAN Idle for z/OSa Link Aggregation for z/VM Layer 2 mode a a. Only in QDIO mode (OSD, OSX) b. Only for FC 1364 and FC 1365 c. Only supported with OSA-Express3 features d. Only for mode OSD e. Only when configured in 1 Gbps and in QDIO mode 128 OSA-Express 10 GbE OSA-Express2 and OSA-Express3 IBM System z Connectivity Handbook 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm 7.3.3 Software support Operating systems that support System z servers also provide support for OSA-Express, including: 򐂰 z/OS All in-service releases of z/OS. For more information go to: http://www.ibm.com/systems/z/os/zos/support/index.html 򐂰 z/VM All in-service releases of z/VM. For more information go to: http://www.vm.ibm.com/techinfo/lpmigr/vmleos.html 򐂰 z/VSE All in-service releases of z/VSE. For more information go to: http://www.ibm.com/servers/eserver/zseries/os/vse/support/support.htm 򐂰 z/Transaction Processing Facility (z/TPF) All in-service releases of z/TPF. For more information go to: http://www.ibm.com/software/htp/tpf/index.html 򐂰 Linux Linux on System z support and downloads. Go to: http://www.ibm.com/servers/eserver/zseries/os/linux/dist.html Note: Certain functionality may require specific levels of an operating system or PTFs; that information is provided when appropriate within this chapter. 7.3.4 Resource Measurement Facility (RMF) RMF (z/OS only) reporting supports the OSA features. This enables you to capture performance data for these features: 򐂰 Microprocessor utilization (per Logical Partition image if it applies) 򐂰 Physical PCI (Peripheral Component Interconnect) bus utilization 򐂰 Bandwidth per port (both read and write directions), per Logical Partition image For example, with this support you can analyze possible bandwidth bottlenecks and perform root cause analysis. 7.4 Summary The OSA features provide direct LAN connectivity as integrated features of the System z. This brings the strengths of System z and z/Architecture to the client/server environment. Table 7-4 summarizes the OSA features as they relate to the different modes of operation and maximum addressing ranges supported by System z servers. Table 7-4 OSA modes of operation and addressing support OSA-Express OSA-Express2 OSA-Express3a 2048 4096 4096 Addresses IP addresses per port (IPv4, IPv6, VIPA) Chapter 7. Open Systems Adapter-Express 129 5444ch08.fm Draft Document for Review July 21, 2010 6:06 pm OSA-Express OSA-Express2 OSA-Express3a Multicast addresses (IPv4 + IPv6) 1024 16384 16384 ARP table size 8192 16384 16384 MAC addresses 2048b 2048 2048 2 2c 2c TCP/IP stacks per port 120 120c 120c SNA PUs per port 4096 4096c 4096c Subchannels per port 240 240c 240c 1 1c 1c 3 3 3 TCP/IP stacks per port 84 / 160 640d 640d Subchannels per port 254 / 480 1920d 1920d 1 / 16 16 16 Non-QDIO (OSE) Subchannels per TCP/IP link CUs per port QDIO (OSD) Subchannels per TCP/IP link CUs per port a. Applies to z196 and z10 only. b. Supported by Ethernet features only (excluding FC 2366). c. 1000BASE-T feature only. d. If multiple priority for queues is enabled, the maximum is reduced to 160 TCP/IP stacks/480 devices. 7.5 References For further information about the OSA features and configuration, see: 򐂰 Open Systems Adapter-Express Customer’s Guide and Reference, SA22-7935 򐂰 Open Systems Adapter-Express Implementation Guide, SG24-5948 򐂰 Communications Server: SNA Network Implementation Guide, SC31-8563 򐂰 Communications Server: IP Configuration, SC31-8513 򐂰 Resource Measurement Facility Report Analysis, SC28-1950 򐂰 IBM Communication Controller Migration Guide SG24-6298 򐂰 IBM Communication Controller for Linux on System z V1.2.1 Implementation Guide, SG24-7223 򐂰 Enterprise Extender Implementation Guide, SG24-7359 130 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch09.fm 8 Chapter 8. OSA-Express Console support This chapter describes the Open Systems Adapter-Express Integrated Console Controller (OSA-ICC) function of the OSA-Express3 and OSA-Express2 1000BASE-T features. The OSA-ICC function supports TN3270 enhancements (TN3270E), local non-SNA DFT 3270 emulation, and 328x printer emulation. This emulation support for console session connections is integrated in System z servers. It can help reduce cost and complexity by eliminating the requirement for external console controllers, such as the IBM 2074 and IBM 3174. The following topics are covered: 򐂰 A description of the OSA-ICC 򐂰 Migration considerations 򐂰 Minimum software support © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 131 5444ch09.fm Draft Document for Review July 21, 2010 6:06 pm 8.1 Description The OSA-Express Integrated Console Controller (OSA-ICC) support is a no-charge function included in Licensed Internal Code (LIC) on z196, z10 and z9 servers. It is available via the OSA-Express3 and OSA-Express2 and 1000BASE-T Ethernet features, and supports Ethernet-attached TN3270E consoles. The OSA-ICC provides a system console function at IPL time and operating systems support for multiple logical partitions. Console support can be used by z/OS, z/OS.e, z/VM, z/VSE, z/TPF, and TPF. The OSA-ICC also supports local non-SNA DFT 3270 and 328x printer emulation for TSO/E, CICS®, IMS, or any other 3270 application that communicates through VTAM. With RPQ 8P2339, 3215 emulation is provided for z/TPF V1R1 and TPF V4R1. With the 1000BASE-T Ethernet features, the OSA-ICC is configured on a port by port basis, using the Channel Path Identifier (CHPID) type OSC. Each port can support up to 120 console session connections, can be shared among logical partitions using Multiple Image Facility (MIF), and can be spanned across multiple Channel Subsystems (CSSs). Figure 8-1 shows an example of the OSA-Express Integrated Console Controller in a single system configuration. LP1 LP2 LP3 LP17 LCSS 0 LP18 LP19 LCSS 1 MIF 1000BASE-T 1000BASE-T Logical partitions 1 and 2 do not have alternate sessions Logical partition 19 sessions use alternate OSC CHPID paths Spanned OSC CHPID PCOMM PCOMM Ethernet Console(s) Subnet 22 Ethernet Subnet 06 Console(s) Figure 8-1 Example of an OSA-ICC configuration The base support for the OSA-ICC includes: 򐂰 Local and remote session connectivity, depending on the provided security and networking environment. 򐂰 Local and remote connections for configuration changes, using security features of the System z Hardware Management Console environment. 򐂰 Coexistence in configurations with IBM 2074 and IBM 3174 console controllers. 132 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch09.fm Migration considerations Operating system console and local non-SNA DFT 3270 terminals have traditionally been supported by IBM 2074 and 3174 control units. Beginning with System z servers, OSA-ICC supersedes these control unit types. Existing System z server configurations implemented with IBM 2074 and 3174 can migrate to OSA-ICC and take advantage of enhanced functionality, flexibility, and cost savings. OSA-ICC can coexist with IBM 2074 and 3174 in a System z configuration. A client workstation can have some TN3270E sessions connecting via OSA-ICC, and other sessions connecting via IBM 2074, at the same time. If your current configuration uses IBM 2074 or 3174 for primary and alternate operating system console control units, you can choose to migrate to an OSA-ICC environment, in a staged process. There are many possible configuration options when planning your migration strategy. One migration strategy would be to: 1. Maintain the existing primary operating system console controls units (IBM 2074 or 3174). 2. Initially implement the OSA-ICC CHPID function to replace the “alternate” operating system console control units. 3. Once you are comfortable with the OSA-ICC function in your environment, swap the OSA-ICC role to primary operating system console support. 4. Use existing IBM 2074 or 3174 control units for “alternate” operating system console support. 5. Eventually replace the “alternate” IBM 2074 or 3174 control units with a second OSA-ICC CHPID implementation. 8.2 Connectivity System z servers have base Licensed Internal Code support for the OSA-ICC function. At least one 1000BASE-T Ethernet feature must installed. The Hardware Management Console (HMC) or Support Element (SE) is used to create the configuration source file for the OSA-ICC CHPID as well as for operation and diagnosis. The OSA Express3 1000BASE-T Ethernet feature (FC 3367 and FC 3369) occupies one I/O slot in the z196 I/O cage or I/O drawer or a z10 I/O cage. FC 3367 has 4 ports, with one CHPID associated with each of the two corresponding ports. FC 3369 has two ports, with one CHPID associated with each of the two corresponding ports. The OSA-Express2 1000BASE-T Ethernet feature (FC 3366) occupies one I/O slot in the z1961 I/O cage or I/O drawer or a z10 or z9 I/O cage. The feature has two independent ports, with one CHPID associated with each port. The OSA-Express 1000BASE-T Ethernet feature (FC 1366) occupies one I/O slot in the z9 I/O cage. The feature has two independent ports, with one CHPID associated with each port. Note: OSA-Express 1000BASE-T Ethernet feature (FC 1366) is no longer orderable. It is carried forward on an upgrade to z9. 1 Only when carried forward Chapter 8. OSA-Express Console support 133 5444ch09.fm Draft Document for Review July 21, 2010 6:06 pm 8.3 Software support Software requirements to support the OSA-ICC function are listed in this section. It includes the operating system levels and PTFs when applicable, as well as TN3270E emulator program support. Refer to the z/OS, z/VM, z/VSE, and z/TPF subsets of the 2098DEVICE, 2097DEVICE, 2096DEVICE, and 2094DEVICE Preventative Service Planning (PSP) buckets as required prior to implementation. Operating systems that support FICON on System z servers include: 򐂰 z/OS All in-service releases of z/OS. For more information go to: http://www.ibm.com/servers/eserver/zseries/zos/support/ 򐂰 z/VM All in-service releases of z/VM. For more information go to: http://www.vm.ibm.com/techinfo/lpmigr/vmleos.html 򐂰 z/VSE All in-service releases of z/VSE. For more information go to: http://www.ibm.com/servers/eserver/zseries/os/vse/support/support.htm 򐂰 z/Transaction Processing Facility (z/TPF) All in-service releases of z/TPF. For more information go to: http://www.ibm.com/software/htp/tpf/index.html 򐂰 Linux For details about Linux on System z support and downloads, go to: http://www.ibm.com/servers/eserver/zseries/os/linux/dist.html TN3270E emulation The TN3270E emulation program used in conjunction with the OSA-ICC must comply with the TN3270E (TN3270 Extensions) protocol (RFC 2355), which allows for the following functions: 򐂰 The ability to request that a connection be associated with a specific 3270 LU or Pool name. 򐂰 Access to the SNA Bind information. Until it receives the Bind information, the session runs at the SSCP-LU level. 򐂰 Handling of SNA positive and negative responses. 򐂰 Universal support of the 3270 ATTN and SYSREQ keys. 򐂰 Support for the emulation of the 328x class of printer. IBM Personal Communications V5.6 or later supports TN3270 Extentions used for OSA-ICC support. Personal Communications is a component of the IBM Access Client Package for Multiplatforms program suite. For more information see: http://www-306.ibm.com/software/network/pcomm/ IBM Tivoli AF/Remote V1.0.1 or later supports TN3270 Extentions used for OSA-ICC support. For more information, see: http://www-306.ibm.com/software/tivoli/products/af-remote/index.html 134 IBM System z Connectivity Handbook 5444ch09.fm Draft Document for Review July 21, 2010 6:06 pm 8.4 Summary Following are the key characteristics of the Open Systems Adapter-Express Integrated Console Controller (OSA-ICC) function: 򐂰 A no charge function integrated in z196, z10, and z9 server Licensed Internal Code (LIC). 򐂰 Supported in new build z196, z10 servers via the OSA-Express3 1000BASE-T feature. 򐂰 Supported in new build z10 and z9 servers via the OSA-Express2 1000BASE-T feature. 򐂰 Supported in z9 servers via the OSA-Express 1000BASE-T feature carried forward from an upgrade. 򐂰 Each feature port can support up to 120 console session connections, be shared among logical partitions, and be spanned across multiple CSSs. 򐂰 Supports TN3270 Enhancements (TN3270E) and local non-SNA DFT 3270 emulation via Ethernet-attached workstations. 򐂰 Console support can be used by z/OS, z/OS.e, z/VM, z/VSE, z/TPF, and TPF. 򐂰 Local non-SNA DFT 3270 and 328x printer emulation support can be used by TSO/E, CICS, IMS, or any other 3270 application that communicates through VTAM. The OSA-ICC function provides a number of potential benefits, such as: 򐂰 Simplicity: – Leverage installed 1000BASE-T Ethernet for both console sessions and networking traffic. – External controllers are no longer needed. – ESCON ports are not required. 򐂰 Scalable capacity: – Facilitates addition of partitions and operations support pools. – Can be shared by up to 60 logical partitions on z196, z10 EC and z9 EC servers. – Can be shared by up to 30 logical partitions on z10 BC and z9 BC servers. 򐂰 Improved availability: – Can enable lights out operation. – Hot plug OSA availability characteristics. – The OSA features are a highly integrated component of the System z server. It has the reliability, availability, and serviceability characteristics inherent in System z. 򐂰 Low operating cost versus external console controller: – Power, cooling, cabling, and floor space. – No rack needed. 8.5 References For further information about the Open Systems Adapter-Express Integrated Console Controller (OSA-ICC) function, see: 򐂰 OSA-Express Integrated Console Controller Implementation Guide, SG24-6364 򐂰 Open Systems Adapter-Express Integrated Console Controller User’s Guide, SA22-7990 򐂰 Input/Output Configuration Program User’s Guide for ICP IOCP, SB10-7037 Chapter 8. OSA-Express Console support 135 5444ch09.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 OSA-Express3 Integrated Console Controller Dual-Port User's Guide, SC23-2266 136 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch10.fm 9 Chapter 9. HiperSockets In this chapter, we provide a high-level overview of HiperSockets as it pertains to the System z servers. The topics covered in this chapter are: 򐂰 HiperSockets overview 򐂰 HiperSockets connectivity 򐂰 HiperSockets functions © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 137 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm 9.1 Description HiperSockets provides the fastest TCP/IP communication between consolidated Linux, z/VM, z/VSE, and z/OS virtual servers on a System z server. HiperSockets provides internal “virtual” Local Area Networks, which act like TCP/IP networks within the System z server. This integrated Licensed Internal Code (LIC) function, coupled with supporting operating system device drivers, establishes a higher level of network availability, security, simplicity, performance, and cost effectiveness than is available when connecting single servers or Logical Partitions together using an external TCP/IP network. The HiperSockets function, also known as internal Queued Direct Input/Output (iQDIO) or internal QDIO, is an integrated function on the System z servers that provides users with attachment to high-speed logical LANs with minimal system and network overhead. HiperSockets eliminates the need to utilize I/O subsystem operations and the need to traverse an external network connection to communicate between Logical Partitions in the same System z server. HiperSockets offers significant value in server consolidation connecting many virtual servers, and can be used instead of certain coupling link configurations in a Parallel Sysplex. HiperSockets is customizable to accommodate varying traffic sizes. Since HiperSockets does not use an external network, it can free up system and network resources, eliminating attachment costs while improving availability and performance. 9.1.1 HiperSockets benefits System z HiperSockets is a technology that provides high-speed TCP/IP connectivity between virtual servers running within different Logical Partitions of a System z server. It eliminates the need for any physical cabling or external networking connections between these virtual servers. The virtual servers form a virtual LAN. Using iQDIO, the communication between virtual servers is through I/O queues set up in the system memory of the System z server. Traffic between the virtual servers is passed at memory speeds. HiperSockets supports multiple virtual LANs, which operate as TCP/IP networks within a System z server. Section 9.2, “Connectivity” on page 142, discusses the number of HiperSockets available for each System z server. HiperSockets is a LIC function of the System z server. It is supported by the following operating systems: 򐂰 򐂰 򐂰 򐂰 All in-service z/OS releases All in-service z/VM releases All in service z/VSE releases Linux on System z There are a number of benefits gained when exploiting the HiperSockets function: 򐂰 HiperSockets can be used to communicate among consolidated servers in a single System z server platform. All the consolidated hardware servers can be eliminated, along with the cost, complexity, and maintenance of the networking components that interconnect them. 򐂰 Consolidated servers that have to access corporate data residing on the System z server can do so at memory speeds, bypassing all the network overhead and delays. 򐂰 HiperSockets can be customized to accommodate varying traffic sizes. (In contrast, LANs like Ethernet and Token Ring have a maximum frame size predefined by their 138 IBM System z Connectivity Handbook 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm architecture.) With HiperSockets, a maximum frame size can be defined according to the traffic characteristics transported for each of the possible HiperSockets virtual LANs. 򐂰 Since there is no server-to-server traffic outside the System z server, a much higher level of network availability, security, simplicity, performance, and cost effectiveness is achieved as compared with servers communicating across an external LAN. For example: – Since HiperSockets has no external components, it provides a very secure connection. For security purposes, servers can be connected to different HiperSockets. All security features, like firewall filtering, are available for HiperSockets interfaces as they are for other TCP/IP network interfaces. – HiperSockets looks like any other TCP/IP interface; therefore, it is transparent to applications and supported operating systems. 򐂰 HiperSockets can also improve TCP/IP communications within a sysplex environment when the DYNAMICXCF facility is used. Server integration with HiperSockets Many data center environments today are multi-tiered server applications, with a variety of middle-tier servers surrounding the System z data and transaction server. Interconnecting that multitude of servers requires the cost and complexity of many networking connections and components. The performance and availability of the inter-server communication is dependent on the performance and stability of the set of connections. The more servers involved, the greater the number of network connections and complexity to install, administer, and maintain. Figure 9-1 shows two configurations. The configuration on the left shows a server farm surrounding a System z server, with its corporate data and transaction servers. There is great complexity involved in the backup of the servers and network connections with this configuration. This environment also causes high administration cost. System z server Multiple external servers HiperSockets z/VM z/OS z/OS Consolidation Linux Guest Systems OSA-Express Many network connections TCP/IP network Few network connections TCP/IP network Figure 9-1 Server consolidation Chapter 9. HiperSockets 139 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm Consolidating that mid-tier workload onto multiple Linux virtual servers running on a System z server requires the very reliable, high-speed network that HiperSockets provides, over which those servers can communicate. In addition, those consolidated servers also have direct high-speed access to database and transaction servers running under z/OS on the same System z server. This is shown in the configuration on the right in Figure 9-1 on page 139. Each consolidated server can communicate with others on the System z server through HiperSockets. In addition, the external network connection for all servers is concentrated over a few high-speed OSA-Express interfaces. 9.1.2 HiperSockets function HiperSockets implementation is based on the OSA-Express Queued Direct Input/Output (QDIO) protocol, hence HiperSockets is called internal QDIO (iQDIO). The LIC emulates the link control layer of an OSA-Express QDIO interface. Typically, before a packet can be transported on an external LAN, a LAN frame has to be built, and the MAC address of the destination host or router on that LAN has to be inserted into the frame. HiperSockets does not use LAN frames, destination hosts, or routers. TCP/IP stacks are addressed by inbound data queue addresses instead of MAC addresses. The System z server LIC maintains a lookup table of IP addresses for each HiperSockets. This table represents a virtual LAN. At the time a TCP/IP stack starts a HiperSockets device, the device is registered in the IP address lookup table with its IP address, and its input and output data queue pointers. If a TCP/IP device is stopped, the entry for this device is deleted from the IP address lookup table. HiperSockets copies data synchronously from the output queue of the sending TCP/IP device to the input queue of the receiving TCP/IP device by using the memory bus to copy the data, via an I/O instruction. The controlling operating system that performs I/O processing is identical to OSA-Express in QDIO mode. The data transfer time is similar to a cross-address space memory move, with hardware latency close to zero. For a data move total elapsed time, the operating system I/O processing time has to be added to the LIC data move time. HiperSockets operations are executed on the processor where the I/O request is initiated by the operating system. HiperSockets starts write operations; the completion of a data move is indicated by the sending side to the receiving side with a Signal Adapter (SIGA) instruction. Optionally, the receiving side can use dispatcher polling instead of handling SIGA interrupts. The I/O processing is performed without using the System Assist Processor (SAP). This implementation is also called thin interrupt. 140 IBM System z Connectivity Handbook 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm The data transfer itself is handled much like a cross-address space memory move using the memory bus, not the System z server I/O bus. Therefore, HiperSockets does not contend with other system I/O activity in the server. Figure 9-2 shows the basic operation of HiperSockets. System z Server Virtual Server 1 Virtual Server 2 TCP/IP TCP/IP Device Driver 3 Virtual Server 3 Device Driver 2 TCP/IP Device Driver 2 2 4/5 1 1 1 Common Lookup Table across entire HiperSockets Figure 9-2 HiperSockets basic operation The HiperSockets operational flow consists of five steps: 1. Each TCP/IP stack registers its IP addresses into a server-wide Common Address Lookup table. There is one lookup table for each HiperSockets virtual LAN. The scope of each LAN is the Logical Partitions defined to share the HiperSockets IQD CHPID. 2. The address of the TCP/IP stack’s receive buffers are appended to the HiperSockets queues. 3. When data is being transferred, the send operation of HiperSockets performs a table lookup for the addresses of the sending and receiving TCP/IP stacks and their associated send and receive buffers. 4. The sending virtual server copies the data from its send buffers into the target virtual server’s receive buffers (System z server memory). 5. The sending virtual server optionally delivers an interrupt to the target TCP/IP stack. This optional interrupt uses the “thin interrupt” support function of the System z server, which means the receiving virtual server will look ahead, detecting and processing inbound data. This technique reduces the frequency of real I/O or external interrupts. Hardware assists A complementary virtualization technology is available for z196, z10 and z9, which includes: 򐂰 QDIO Enhanced Buffer-State Management (QEBSM): Two hardware instructions designed to help eliminate the overhead of hypervisor interception 򐂰 Host Page-Management Assist (HPMA): An interface to the z/VM main storage management function designed to allow the hardware to assign, lock, and unlock page frames without z/VM hypervisor assistance These hardware assists allow a cooperating guest operating system to initiate QDIO operations directly to the applicable channel, without interception by z/VM, thereby helping to provide additional performance improvements. Support is integrated in System z. However, you should always check the appropriate subsets of the 2817DEVICE, 2098DEVICE, Chapter 9. HiperSockets 141 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm 2097DEVICE, 2096DEVICE, and 2094DEVICE Preventative Service Planning (PSP) buckets prior to implementation. 9.2 Connectivity HiperSockets has no external components or external network. There is no internal or external cabling. The HiperSockets data path does not go outside the System z server platform. HiperSockets is not allocated a CHPID until it is defined, and it does not take an I/O cage nor an I/O drawer slot. If you have used all of the available CHPIDs on the System z server, you cannot enable HiperSockets; therefore, HiperSockets must be included in your overall channel I/O planning. HiperSockets TCP/IP devices are configured similar to OSA-Express QDIO devices. Each HiperSockets requires the definition of a CHPID like any other I/O interface. The CHPID type for HiperSockets is IQD, and the CHPID number must be in the range from hex 00 to hex FF. No other I/O interface can use a CHPID number defined for a HiperSockets, even though HiperSockets does not occupy any physical I/O connection position. Real LANs have a maximum frame size limit defined by their architecture. The maximum frame size for Ethernet is 1492 bytes, and for Gigabit Ethernet there is the jumbo frame option for a maximum frame size of 9 kilobytes (KB). The maximum frame size for a HiperSockets is assigned when the HiperSockets CHPID is defined. Frame sizes of 16 KB, 24 KB, 40 KB, and 64 KB can be selected. The default maximum frame size is 16 KB. The selection depends on the characteristics of the data transported over a HiperSockets, and is also a trade-off between performance and storage allocation. The MTU size used by the TCP/IP stack for the HiperSockets interface is also determined by the maximum frame size. These values are shown in Table 9-1. Table 9-1 Maximum frame size and MTU size Maximum frame size Maximum transmission unit size 16 KB 8 KB 24 KB 16 KB 40 KB 32 KB 64 KB 56 KB The maximum frame size is defined in the hardware configuration (IOCP), using the CHPARM for System z. z/OS allows the operation of multiple TCP/IP stacks within a single image. The read control and write control I/O devices are required only once per image, and are controlled by VTAM. Each TCP/IP stack within the same z/OS image requires one I/O device for data exchange. Running one TCP/IP stack per Logical Partition, z/OS requires three I/O devices (as do z/VM and Linux). Each additional TCP/IP stack in a z/OS Logical Partition requires only one additional I/O device for data exchange. The I/O device addresses can be shared between z/OS systems running in different Logical Partitions. Therefore, the number of I/O devices will not be a limitation for z/OS. 142 IBM System z Connectivity Handbook 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm An IP address is registered with its HiperSockets interface by the TCP/IP stack at the time the TCP/IP device is started. IP addresses are removed from an IP address lookup table when a HiperSockets device is stopped. Under operating system control, IP addresses can be reassigned to other HiperSockets interfaces on the same HiperSockets LAN. This allows flexible backup of TCP/IP stacks. Note that Reassignment is only possible within the same HiperSockets LAN. A HiperSockets is one network or subnetwork. Reassignment is only possible for the same operating system type. For example, an IP address originally assigned to a Linux TCP/IP stack can only be reassigned to another Linux TCP/IP stack, a z/OS dynamic VIPA can only be reassigned to another z/OS TCP/IP stack, or a z/VM TCP/IP VIPA can only be reassigned to another z/VM TCP/IP stack. The LIC performs the reassignment in force mode. It is up to the operating system’s TCP/IP stack to control this change. Enabling HiperSockets requires the definition of a CHPID defined as type=IQD using HCD and IOCP. This CHPID is treated like any other CHPID, and is counted as one of the available channels within the System z server. The HiperSockets LIC on System z servers supports: 򐂰 Up to 32 independent HiperSockets on z196 and 16 independent HiperSockets on z10 and z9 servers. 򐂰 For z/OS, z/VM, Linux, and z/VSE, the maximum number of TCP/IP stacks or HiperSockets communication queues that can concurrently connect on a single System z server is 4096. 򐂰 Maximum total of 12288 I/O devices (valid subchannels) across all HiperSockets 򐂰 Maximum total of 12288 IP addresses across all HiperSockets These IP addresses include the HiperSockets interface, as well as Virtual IP addresses (VIPA) and dynamic Virtual IP Addresses (DVIPA) that are defined to the TCP/IP stack. With the introduction of the channel subsystem, transparent sharing of HiperSockets is possible with the extension to the Multiple Image facility (MIF). HiperSockets channels can be configured to multiple Channel Subsystems (CSS). They are transparently shared by any or all of the configured logical partitions without regard to the CSS to which the partition is configured. Figure 9-3 reflects spanned HiperSockets defined on a System z server. For more information about spanning, refer to 2.1.8, “Channel spanning” on page 21. Chapter 9. HiperSockets 143 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm z/VM LP1 z/OS Linux LP14 LP15 z/VM LP 17 CSS 0 MIF 1 MIF 2 CHPID CHPID CHPID 00 01 02 CHPID FF PCHID PCHID PCHID 10D 10B 10C Linux LP18 LP30 CSS 1 MIF F CHPID 03 Share Linux PCHID 20A MIF 1 CHPID 04 SPAN HiperSockets CHPID 03 MIF 2 MIF 3 MIF F CHPID CHPID 00 01 CHPID CHPID CHPID 05 22 FF Share PCHID PCHID 245 246 PCHID PCHID 248 249 HiperSockets CHPID 05 HiperSockets CHPID 04 Figure 9-3 Spanned and non-spanned HiperSockets defined 9.3 Supported functions This section discusses additional functions that are supported by HiperSockets. 9.3.1 Broadcast support Broadcasts are now supported across HiperSockets on Internet Protocol Version 4 (IPv4) for applications. Now, applications that use the broadcast function can propagate the broadcast frames to all TCP/IP applications that are using HiperSockets. This support is applicable to Linux, z/OS, and z/VM environments. 9.3.2 VLAN support Virtual Local Area Networks (VLANs), IEEE standard 802.1q, are supported by Linux on System z and z/OS V1R8 (or later) for HiperSockets. VLANs can reduce overhead by allowing networks to be organized by traffic patterns rather than physical location. This enhancement permits traffic flow on a VLAN connection both over HiperSockets and between HiperSockets and OSA-Express Ethernet features. 9.3.3 IPv6 support HiperSockets supports Internet Protocol Version 6 (IPv6). IPv6 is the protocol designed by the Internet Engineering Task Force (IETF) to replace Internet Protocol Version 4 (IPv4) to help satisfy the demand for additional IP addresses. 144 IBM System z Connectivity Handbook 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm The support of IPv6 on HiperSockets (CHPID type IQD) is exclusive to System z servers, and is supported by z/OS and z/VM. IPv6 support is currently available on the OSA-Express3, OSA-Express2, and OSA-Express features in the z/OS, z/VM, and Linux on System z environments. HiperSockets support of IPv6 (CHPID type IQD) on System z servers requires, at a minimum: 򐂰 z/OS V1.7 or later 򐂰 z/VM V5.2 with PTFs or later Support of guests is expected to be transparent to z/VM if the device is directly connected to the guest (pass through). 9.3.4 HiperSockets Network Concentrator Traffic between HiperSockets and OSA-Express can be transparently bridged using the HiperSockets Network Concentrator, without requiring intervening network routing overhead, thus increasing performance and simplifying the network configuration. This is achieved by configuring a connector Linux system that has HiperSockets and OSA-Express connections defined. The HiperSockets Network Concentrator registers itself with HiperSockets as a special network entity to receive data packets destined for an IP address on the external LAN via an OSA port. The HiperSockets Network Concentrator also registers IP addresses to the OSA feature on behalf of the TCP/IP stacks using HiperSockets, thus providing inbound and outbound connectivity. HiperSockets Network Concentrator support is performed using the next-hop-IP-address in the Queued Direct Input/Output (QDIO) header, instead of using a Media Access Control (MAC) address. Therefore, VLANs in a switched Ethernet fabric are not supported by this support. TCP/IP stacks using only HiperSockets to communicate among one another with no external network connection see no difference, and the HiperSockets support and the networking characteristics are unchanged. To exploit HiperSockets Network Concentrator unicast and multicast support, a Linux distribution is required. s390-tools are needed as well; they can be obtained from the developerWorks® site at: http://www10.software.ibm.com/developerworks/opensource/linux390/index.shtml 9.3.5 HiperSockets Layer 2 support HiperSockets supports two transport modes on the z196 and z10 EC: Layer 2 (Link Layer) and Layer 3 (Network and IP Layer). HiperSockets is protocol independent and supports the following traffic types: 򐂰 Internet Protocol (IP) Version 4 or Version 6 򐂰 Non-IP (such as AppleTalk, DECnet, IPCX, NetBIOS, SNA, or others) Each HiperSockets device has its own Layer 2 MAC address and allows the use of applications that depend on a Layer 2 address such as DHCP servers and firewalls. LAN administrators can configure and maintain the mainframe environment in the same fashion as they do in other environments. This eases server consolidation and simplifies network configuration. The HiperSockets device performs automatic MAC address generation to create uniqueness within and across logical partitions and servers. MAC addresses can be locally administered, and the use of Group MAC addresses for multicast and broadcasts to all other Layer 2 devices on the same HiperSockets network is supported. Datagrams are only delivered Chapter 9. HiperSockets 145 5444ch10.fm Draft Document for Review July 21, 2010 6:06 pm between HiperSockets devices using the same transport mode (Layer 2 with Layer 2 and Layer 3 with Layer 3). A HiperSockets device may filter inbound datagrams by VLAN identification, the Ethernet destination MAC address, or both. This reduces the amount of inbound traffic, leading to lower CPU utilization by the operating system. As with Layer 3 functions, HiperSockets Layer 2 devices can be configured as primary or secondary connectors or multicast routers enabling high performance and highly available Link Layer switches between the HiperSockets network and an external Ethernet. HiperSockets Layer 2 support is exclusive to z196 and z10 servers and is supported by Linux on System z, and by z/VM guest exploitation. 9.3.6 HiperSockets multiple write facility HiperSockets performance has been increased by allowing streaming of bulk data over a HiperSockets link between logical partitions. The receiving partition can process larger amounts of data per I/O interrupt. The improvement is transparent to the operating system in the receiving partition. Multiple writes with fewer I/O interrupts reduces CPU utilization of both the sending and receiving logical partitions and is supported in z/OS. 9.3.7 zIIP-Assisted HiperSockets for large messages In z/OS, HiperSockets has been enhanced for zIIP exploitation. Specifically, the z/OS Communications Server allows the HiperSockets Multiple Write Facility processing for outbound large messages originating from z/OS to be performed on a zIIP. z/OS application workloads based on XML, HTTP, SOAP, Java, as well as traditional file transfer, can benefit from zIIP enablement by lowering general purpose processor utilization. When the workload is eligible, the HiperSockets device driver layer processing (write command) is redirected to a zIIP, which will unblock the sending application. zIIP-Assisted HiperSockets for large messages is available only on zEnterprise 196 and System z10 servers with z/OS V1R10 (plus PTF UK37306) or above. 9.3.8 HiperSockets Network Traffic Analyzer HiperSockets Network Traffic Analyzer (HS NTA) is a function available in the z196 and System z10 Licensed Internal Code (LIC). It can make problem isolation and resolution simpler, by allowing Layer 2 and Layer 3 tracing of HiperSockets network traffic. HS NTA permits Linux on System z to control tracing of the internal virtual LAN. It captures records into host memory and storage (file systems) that can be analysis by system programmers and network administrators, using Linux on System z tools to format, edit, and process the trace records. A customized HiperSockets NTA rule enables you to authorize an LPAR to trace messages only from LPARs that are eligible to be traced by the NTA on the selected IQD channel. HS NTA rules can be set up on the Support Element (SE). There are four different types of rules for the HS NTA: 1. Tracing is disabled for all IQD channels in the system (the default rule) 2. Tracing is disabled for a specific IQD channel 146 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch10.fm 3. Tracing is allowed for a specific IQD channel (all LPARS can be set up for NTA and all LPARs are eligible to be traced by an active Network Traffic Analyzer) 4. Customized tracing is allowed for a specific IQD channel 9.4 Summary HiperSockets is part of z/Architecture technology, and includes QDIO and advanced adapter interrupt handling. The data transfer itself is handled much like a cross address space memory move, using the memory bus. Therefore, HiperSockets does not contend with other system I/O activity in the server. HiperSockets can be defined to separate traffic between specific virtual servers and Logical Partitions on one System z server. Virtual Private Networking or network Virtual LANs across HiperSockets are supported to further isolate traffic as required. With integrated HiperSockets networking, there are no server to server traffic flows outside the System z server, thus no exposure to physical wire tapping or probes on the LAN. The z196 supports up to 32 HiperSockets, while z10 and z9 servers support up to 16 HiperSockets, and spanned channel support allows sharing of HiperSockets across multiple Channel SubSystems (CSSs) and LPARs. 9.5 References For further information about the HiperSockets function and configuration, refer to HiperSockets Implementation Guide, SG24-6816. A HiperSockets Network Traffic Analyzer (HS NTA) "Frequently Asked Questions document can be found at: http://www-01.ibm.com/common/ssi/apilite?infotype=SA&infosubt=ST&lastdays=1825&hit limit=200&ctvwcode=US&pubno=ZSQ*USEN&appname=STG_ZS_USEN_FQ&additional=summary&con tents=keeponlit Chapter 9. HiperSockets 147 5444ch10.fm 148 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444ch11.fm 10 Chapter 10. Coupling links and common time This chapter describes the connectivity options that support clustering technology (known as Parallel Sysplex) and common time on System z servers. The following topics are covered: 򐂰 򐂰 򐂰 򐂰 Cluster technology Server Time Protocol Connectivity options Time functions © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 149 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm 10.1 System z cluster technology Clustering technology is designed to bring the power of parallel processing to business-critical applications. A Parallel Sysplex cluster consists of up to 32 z/OS images, coupled to one or more Coupling Facilities (CF), using high-speed specialized links for communication and time keeping. The Coupling Facilities, at the heart of the cluster, enable high-speed, record-level read/write data sharing among the images in a cluster. When configured properly, a cluster has no single point of failure and can provide users with near-continuous application availability over planned and unplanned outages. Figure 10-1 shows a configuration of a Parallel Sysplex with IBM Sysplex Timers. A Sysplex Timer provides a common time reference to all z/OS images, which assists in managing the cluster of multiple server footprints as a single operational image. The common time source also enables proper sequencing and time stamping of updates to shared databases, a feature critical to the recoverability of the shared data. Coupling Facility Coupling Facility Sysplex Timer System z Server 11 12 1 10 2 9 3 8 4 7 6 5 11 12 1 10 2 9 3 8 4 7 6 5 FICON Director 1 DASD Figure 10-1 Parallel Sysplex using stand-alone CFs and Sysplex Timers 150 IBM System z Connectivity Handbook System z Server FICON Director n DASD 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Figure 10-2 shows a Parallel Sysplex configuration with the Server Time Protocol (STP) feature. STP is designed to provide time synchronization for multiple servers and Coupling Facilities, without requiring a Sysplex Timer. The use of Network Time Protocol servers as an External Time Source (ETS) is supported by STP. Note: A time synchronization mechanism, either IBM Sysplex Timer or Server Time Protocol (STP), is a mandatory hardware requirement for a Parallel Sysplex environment consisting of more than one server. Coupling Facility 2 Coupling Facility 1 Arbiter System z Server System z Server Preferred Time Server Backup Time Server FICON Director 1 FICON Director n DASD DASD Figure 10-2 Parallel Sysplex using stand-alone CFs and STP STP is a server-wide facility that is implemented in the Licensed Internal Code (LIC) of System z servers and CFs, for presenting a single view of time to PR/SM™. It is a message-based protocol in which timekeeping information is passed over coupling links between servers. The servers are configured with coupling peer mode links, such as the InfiniBand coupling links (PSIFB), Integrated Cluster Bus peer (ICB-4) links, Inter-System Channel peer (ISC-3) link, and the Internal Coupling peer (IC) channel. All of these, except the IC channel, are supported by STP. With these components, coupling performance is balanced with the performance of z/OS images running on the System z servers in a common time cluster. A Coupling Facility (CF) runs the Coupling Facility Control Code (CFCC) that is loaded into main storage at Power On Reset (POR) time. CFCC can run on a Stand-Alone CF server or in a logical partition. Chapter 10. Coupling links and common time 151 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm 10.1.1 Coupling links and STP STP is a message-based protocol in which STP timekeeping information is passed over externally defined coupling links. Coupling links that can be used to transport STP messages are the InfiniBand coupling links (PSIFB), InterSystem Channel-3 (ISC-3) links in peer mode, and the Integrated Cluster Bus 4 (ICB-4) links. There are a number of advantages to using the PSIFB, ISC-3, and ICB-4 links to exchange STP messages. 򐂰 By using the same links to exchange timekeeping information and Coupling Facility messages in a Parallel Sysplex, STP can scale with distance. Servers exchanging messages over short links, such as ICB-4, and servers exchanging messages over medium length links, such as PSIFB links, can now meet more stringent synchronization requirements than servers exchanging messages over long PSIFB LR and ISC-3 links (distances up to 100 km). This is an enhancement over the Sysplex Timer implementation, which does not scale with distance. 򐂰 Coupling links also provide the connectivity needed in a Parallel Sysplex. Therefore, there is a potential benefit of minimizing the number of cross-site links required in a multi-site cluster. Timing-only coupling links For a server that is not part of a Parallel Sysplex, but required to be time synchronized, coupling links must be configured in order for the server to be configured in a CTN. HCD has been enhanced to support timing-only links to be defined when a CF does not exist at either end of the coupling link. The timing-only links can be of type CIB for InfiniBand coupling links, CFP for ISC-3 in peer mode, or CBP for ICB-4. The control unit type is “STP” and has no devices defined to it. Note: CF messages cannot be transferred over timing-only links. Coupling link redundancy for STP Between any two servers that are intended to exchange STP messages, it is recommended that each server be configured such that at least two coupling links exist for communication between the servers. This prevents the loss of one link causing the loss of STP communication between the servers. If a server does not have a CF logical partition, timing-only links can be used to provide STP connectivity. The maximum number of attached servers supported by any STP-configured server in a Coordinated Timing Network is equal to the maximum number of coupling links supported by the server in the configuration. For more details refer to Server Time Protocol Planning Guide, SG24-7280, and Server Time Protocol Implementation Guide, SG24-7281. 10.1.2 Multi-site sysplex considerations If a sysplex is configured across two or more sites, it is necessary to synchronize the servers. Consequently, you must plan to extend the PSIFB LR or ISC-3 links beyond the distance supported without repeaters. IBM supports Wavelength Division Multiplexing (WDM) products qualified by IBM for usage in multi-site sysplex solutions such as GDPS. If any STP messages or Sysplex Timer signals are to be transmitted across multiple sites through WDM products, ensure that only qualified WDM products and supported configurations are used. 152 IBM System z Connectivity Handbook 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm For a list of WDM vendor products that are qualified for Sysplex Timer links, PSIFB LR, and ISC-3 links (transporting CF or STP messages) in a multi-site sysplex, refer to 11.6.2, “System z qualified WDM vendor products” on page 183. 10.2 Connectivity When configuring a common time cluster—from a hardware perspective—connectivity is a primary area of focus, as are other basic hardware components, such as the CF, Sysplex Timer or Server Time Protocol, and coupling links. In a Parallel Sysplex, the objective is any-to-any connectivity and nondisruptive planned or unplanned outages. It is essential that all channels and Directors be included in the design. The primary reasons for this are to ensure cross-connectivity to all devices in the cluster, and to potentially eliminate the effects of any outages. For availability purposes, we recommend that there be at least two coupling links connecting each server to each CF in a Parallel Sysplex cluster. The performance, availability, and distance requirements of a Parallel Sysplex environment are the key factors that will identify the appropriate connectivity option for a given configuration. 10.2.1 Coupling link options To ensure that sysplex environments continue to scale with today’s efficiency, as server performance grows, coupling link design is continuously enhanced. ISC and ICB links on System z only operate in peer mode, as defined in HCD/IOCP. Peer mode supports coupling between System z servers and provides both sender and receiver capability on the same link. Peer links provide up to seven expanded buffer sets. On the z196, z10 and z9 servers InfiniBand coupling links (PSIFB) are available. PSIFB supports peer mode connectivity between: 򐂰 Any z196 or z10 to any z9 (Link is 12x IB-DDR but operates at SDR) 򐂰 Any z196 or z10 to any z10 or z196 (12x IB-DDR and 1x IB-SDR or DDR) 򐂰 z9 to z9 PSIFB is not supported Chapter 10. Coupling links and common time 153 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Supported connections, illustrated in Figure 10-3, include: 򐂰 From System z general purpose models to System z coupling facilities 򐂰 From System z general purpose models to non-System z coupling facilities 򐂰 Within a System z server, an Internal Coupling (IC) link uses memory-to-memory data transfers between logical partitions (LPs). z9 EC/z9 BC Server z/OS Sysplex LPs CF01 ICF ISC-3 ICB-4 ICB-3 IFB ICB-4 IFB z10 EC/z10 BC Server z9 EC/z9 BC Server ICB-4 z/OS Sysplex LPs IFB z/OS IC Sysplex LPs z196 Server ISC-3 IFB z/OS CF02 ICF ESCON / FICON DASD DASD Figure 10-3 Parallel Sysplex connectivity options 154 IBM System z Connectivity Handbook DASD Sysplex LPs 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Table 10-1 shows the coupling link support for each System z server. Table 10-1 Coupling link server support Link type Feature code Maximum supported linksa z196 z10 EC z10 BC z9 EC z9 BC IC 32 32 32 32 32 ISC-3 48 48 48 48 48 ICB-4 n/a 16 12 16 16b ICB-3 n/a n/a n/a 16 16 HCA2-O LR (PSIFB LR) 32c 32d 12 n/a n/a HCA2-O (PSIFB) 32c 32d 12 n/a n/a HCA1-O (PSIFB) n/a n/a n/a 16e 12 a. For z196 servers the maximum number of coupling links (CHPIDs) combined (PSIFB, active ISC-3 links, and IC) is 128 and up to 80 physical external links (PSIFB, active ISC-3). For z9 or z10 servers the maximum number of coupling links (CHPIDs) combined (ICB-3, ICB-4, PSIFB, active ISC-3 links, and IC) is 64. b. A maximum of eight ICB-4s are supported with a z9 BC capacity setting A01. c. A maximum of 16 PSIFB links are supported on z196 model M15. d. A maximum of 16 PSIFB links are supported on z10 EC model E12. e. A maximum of 12 PSIFB links are supported on z9 EC model S08. Note: System z9 is the last server to support ICB-3 connections. System z10 is the last server to support ICB-4 connections. zEnterprise 196 is the last server to offer ordering of ISC-3 connections. For information about distance support for coupling links see Chapter 11, “Extended distance solutions” on page 167. 10.2.2 Internal Coupling (IC) link IC links are Licensed Internal Code-defined links to connect a CF to a z/OS logical partition in the same server. These links are available on all System z servers. The IC link is a System z server coupling connectivity option that enables high-speed, efficient communication between a CF partition and one or more z/OS logical partitions running on the same server. The IC is a linkless connection (implemented in Licensed Internal Code) and so does not require any hardware or cabling. An IC link is a fast coupling link, using memory-to-memory data transfers. IC links do not have PCHID numbers, but do require CHPIDs. IC links require an ICP channel path definition at the z/OS and the CF end of a channel connection to operate in peer mode. They are always defined and connected in pairs. The IC link operates in peer mode and its existence is defined in HCD/IOCP. IC links have the following attributes: 򐂰 On System z servers, IC links operate in peer mode (channel type ICP). 򐂰 Provide the fastest connectivity, significantly faster than any external link alternatives. 򐂰 Result in better coupling efficiency than with external links, effectively reducing the server cost associated with Parallel Sysplex technology. Chapter 10. Coupling links and common time 155 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 Can be used in test or production configurations, and reduce the cost of moving into Parallel Sysplex technology while enhancing performance and reliability. 򐂰 Can be defined as spanned channels across multiple CSSs. 򐂰 Are free of charge (no feature code). Employing ICFs with IC channels will result in considerable cost savings when configuring a cluster. IC links are enabled by defining channel type ICP. A maximum of 32 IC channels can be defined on a System z server. 10.2.3 InterSystem Channel-3 (ISC-3) ISC-3 provides the connectivity required for data sharing between the CF and the systems. These links also provide connectivity for STP messaging between any z196, z10, or z9 server and CF. ISC links support point-to-point connections (directly connecting CFs and systems), and require a unique channel definition at each end of the link. The ISC-3 feature consists of a Mother card (ISC-M feature code 0217), and one or two ISC-3 Daughter cards (ISC-D feature code 0218). Each ISC-3 Daughter card has two independent ports with one CHPID associated with each active port. A port on an ISC-3 Daughter card is activated using Licensed Internal Code - Control Code (LIC-CC) via feature code 0219. The feature codes for ISC-M and ISC-D cards are not orderable. Instead, ISC links are ordered using the ISC port activation feature code 0219. The System z configurator build process supplies the appropriate number of ISC-M and ISC-D cards based on the number of port activation features ordered. The ports are activated in increments of one. Extra ISC-M cards can be ordered, up the maximum allowed per server or the number of ISC-D cards required to fulfill the ISC-3 link order, whichever is smaller. Each ISC-3 port operates peer mode. The ISC-3 port mode is defined by the CHPID type (CFP) via HCD/IOCP. ISC-3 feature ports in peer mode provide connection from general purpose models to coupling facilities. See Figure 10-3 on page 154. An ISC-3 port is defined in peer mode when its associated CHPID is defined as CHPID type CFP. Each feature port supports connection to another server’s ISC-3 port via 9 micron single mode fiber optic cable terminated with an LC connector. An ISC-3 port in peer mode operates at 2 Gbps and supports a maximum unrepeated distance of 10 km. RPQ 8P2197 (ISC-3 Long Distance Option) provides an ISC-3 Daughter card with two active ports that clock at 1 Gbps and supports a maximum unrepeated distance of 20 km. Under certain conditions, RPQ 8P2263 (or RPQ 8P2340 for the z10 BC) may be required in conjunction with RPQ 8P2197. Check with your IBM representative. See Table 11-1 on page 168 for more information about unrepeated distance support. ISC-3 in peer mode supports STP. Note: The zEnterprise 196 is the last server to offer ordering of ISC-3 connections. 156 IBM System z Connectivity Handbook 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm 10.2.4 Integrated Cluster Bus Integrated Cluster Bus links are members of the family of coupling link options available on System z10 and previous System z servers. They are faster than ISC links, attaching directly to a Self-Timed Interconnect (STI) bus of the server. The ICB features are highly integrated, with very few components, and provide better coupling efficiency (less server overhead associated with coupling systems) than ISC-3 links. They are an available method for coupling connectivity when connecting System z10 and previous System z servers over short distances (seven meters). For longer distances, PSIFB (up to 150 meter), PSIFB LR (up to 100 km), or ISC-3 links (up to 100 km) must be used. Note: ICB features use 10 meter (33 foot) copper cables. Approximately three meters of cable length is used for internal routing and cable strain relief within the servers. This requires the maximum distance between the two servers’ raised floor entry cutouts to be no greater than seven meters. Integrated Cluster Bus-4 (ICB-4) The Integrated Cluster Bus-4 (ICB-4) feature (FC 3393) is available on System z9 and z10 servers. The ICB-4 feature operates at 2 GBps in peer mode only and is defined as CHPID type CBP via HCD/IOCP. The z10 and z9 servers’ ICB-4 feature connection consists of one link that attaches directly to a 2 GBps STI port in the z10, or z9 STI connector in the CEC cage or CPC drawer on z10 BC. It does not require connectivity via a card in the server I/O cage. The ICB-4 link connection is made via a unique 10 meter (33 foot) copper cable (FC 0228), which connects directly to an STI port on System z at each end of the link. This restricts the distance between two servers’ raised floor entry cutouts to 7m due to cable routing. For longer distances, InfiniBand coupling links or ISC-3 peer mode links must be used. The z10 EC and z9 servers support up to 16 ICB-4 links, while the z10 BC servers support up to 12 ICB-4 links. ICB-4 supports STP. Note: The zEnterprise 196 does not support ICB-4 links. Integrated Cluster Bus-3 (ICB-3) Integrated Cluster Bus-3 (ICB-3) feature (FC 0993) is available on z9 EC and z9 BC servers. The ICB-3 feature operates at 1 GBps in peer mode only and is defined as CHPID type CBP via HCD/IOCP. The ICB-3 link connection is made via a unique 10 meter (33 foot) copper cable (FC 0227), which connects directly to a 1 Gbps ICB-3 port on the System z server STI Extender card or directly to the server at each end of the link. This restricts the distance between servers’ raised floor entry cutouts to 7 meters due to cable routing. One ICB-3 feature is required on the server at each end of the link. For longer distances ISC-3 peer mode links or InfiniBand coupling links must be used. The same ICB-3 cable type (FC 0227) connects all servers supporting ICB-3 connections. The ICB-3 feature port on the z9 servers is implemented as follows: The ICB-3 feature (FC 0993) connection is via an STI-3 Extender card which occupies one slot in the I/O cage. The STI-3 Extender card has two ICB-3 ports, with one PCHID associated with each port. The STI-3 Extender card converts its 2 GBps STI input into two 1 Gbps ICB-3 ports. Up to 16 ICB-3 links are supported on System z9 servers. The configurator build process supplies the appropriate number of STI-3 Extender cards based Chapter 10. Coupling links and common time 157 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm on the number of ICB-3 feature ports ordered on System z9. The ports are ordered in increments of one. ICB-3 supports STP on System z9 BC. Note: System z9 is the last family of servers to support ICB-3 connections. ICB connectivity options Table 10-2 summarizes the ICB connectivity between System z10 and System z9. System z196 does not support ICB connections. Table 10-2 ICB connectivity options Coupled server ICB links z10 ICB-4 (2 GBps) z9 ICB-4 (2 GBps) z9 ICB-3 (1 GBps) z10 ICB-4 (2 GBps) Yes Yes n/a z9 ICB-4 (2 GBps) Yes Yes n/a z9 ICB-3 (1 GBps) n/a n/a Yes 10.2.5 InfiniBand coupling links InfiniBand coupling links (PSIFB) are high speed links on z196, z10 and z9 servers. The PSIFB coupling links originate from three types of fanouts: 򐂰 HCA2-O (FC 0163) 򐂰 HCA2-O LR (FC 0168) 򐂰 HCA1-O (FC 0167) The HCA2-O fanouts support InfiniBand Double Data Rate (IB-DDR) and InfiniBand Single Data Rate (IB-SDR) optical links. 򐂰 The HCA2-O fanout supports PSIFB coupling links at distances of up to 150 meters. PSIFB coupling links operate at 6 GBps (12x IB-DDR) when connecting a z196 or z10 to z196 and z10 servers, and at 3 GBps (12x IB-SDR) when connecting a z196 or z10 to a z9. The link speed is auto-negotiated to the highest common rate. Note: A System z9 uses an HCA1-O (12x IB-SDR) fanout when connected to a z196 or z10 server. 򐂰 The HCA2-O LR fanout supports PSIFB Long Reach (PSIFB LR) coupling links for distances of up to 10 km and up to 100 km when repeated through a DWDM. This fanout is supported on z196 and z10. PSIFB LR coupling links operate at up to 5.0 Gbps (1x IB-DDR) between z196 and z10 servers, or automatically scale down to 2.5 Gbps (1x IB-SDR) depending on the capability of the attached equipment. The HCA1-O fanout supports InfiniBand Single Data Rate (IB-SDR) optical links. The HCA1-O fanout supports PSIFB coupling links at distances of up to 150 meters. PSIFB coupling links operate at 3 GBps (12x IB-SDR) when connecting the z9 server to a z196 or z10 server. PSIFB coupling links of either type are defined as CHPID type CIB in HCD/IOCP. 158 IBM System z Connectivity Handbook 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm PSIFB coupling links PSIFB coupling links use a fiber optic cable that is connected to a Host Channel Adapter fanout in the z196, z10, or z9 server. The maximum distance for a PSIFB link is 150 meters (492 feet). The fiber cables are industry standard OM3 50/125 micrometer multimode optical cables with Multifiber Push-On (MPO) connectors. Up to 32 PSIFB coupling links are supported in a z196 and z10 EC server, up to 12 in a z10 BC server, and up to 16 are supported on a System z9. PSIFB LR coupling links PSIFB LR coupling links use a fiber optic cable that is connected to a Host Channel Adapter fanout in the z196 or z10 server. The maximum unrepeated distance for a PSIFB LR link is 10 km; when using repeaters the maximum distance is up to 100 km. The fiber cables used for PSIFB LR links are standard 9µm SM cables with an LC Duplex connector. Up to 32 PSIFB LR coupling links are supported in a z196 and z10 EC server and up to 12 are supported in a z10 BC server. Each HCA has two ports supporting two coupling links. One HCA2-O LR fanout is required at each end of the link. Table 10-3 shows the supported PSIFB connectivity options for System z servers. Table 10-3 PSIFB connectivity options with appropriate data link rates Coupled server PSIFB links z196 or z10 z9 z196 or z10 z9 PSIFB PSIFB LR PSIFB 6 GBps - 3 GBps - 2.5 or 5 Gbps - 3 GBps - - Note: The InfiniBand link data rate of 6 GBps, 3 GBps, 5 Gbps, or 2.5 Gbps does not represent the performance of the link. The actual performance depends on many factors, including latency through the adapters, cable lengths, the type of workload, and so forth. With InfiniBand coupling links, while the link data rate may be higher than that of ICB links, the service times of coupling operations are greater. Up to a total of 16 HCA2-O and HCA2-O LR fanouts are supported on z196 and z10 EC servers, up to 6 fanouts on z10 BC, and up to 8 HCA1-O fanouts on System z9 servers (the maximum is 6 for model S08 and z9 BC). These fanouts provide a maximum of 32 ports for coupling links on z196 and z10 EC servers, 12 ports for coupling links on z10 BC, and 16 ports on z9 servers. Each fanout has an optical transmitter and receiver module and allows dual simplex operation. A fanout has two ports for optical link connections and supports up to 16 CHPIDs across both ports. These CHPIDs can be defined in IOCDS for external coupling links and require a fiber optic cable to connect to another server or the same server. Note: We recommend not to define more than 8 CHPIDs (combined) across the two ports. Chapter 10. Coupling links and common time 159 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Each fanout used for coupling links has an assigned Adapter ID number (AID) that must be used for definitions in IOCDS to have a relation between the physical fanout location and the CHPID number. The STP-only coupling link is supported on PSIFB links. Fanout Adapter ID (AID) Unlike channels installed in an I/O cage or I/O drawer, which are identified by a PCHID number related to their physical location, coupling link fanouts and ports are identified by an Adapter ID (AID). The Adapter ID value depends on its physical location. The AID must be used to assign a CHPID to the fanout in the IOCDS definition. The CHPID assignment is done by associating the CHPID to an AID and port. Table 10-4 identifies the AID assignment for each fanout slot in relation to location on a z196 and z10 EC new build. Table 10-4 AID number assignment for z196 and z10 EC Book Slot Fanout slot AIDs First 6 D1, D2, D5-DA 08-0F Second 15 D1, D2, D5-DA 18-1F Third 10 D1, D2, D5-DA 10-17 Fourth 1 D1, D2, D5-DA 00-07 Top to bottom the fanout slots are numbered D1 to DA, as shown in Table 10-5. Slots D3 and D4 have no fanouts installed. Note: Slots D1 and D2 are not used in a 4-book z196 or z10 EC server, and only partially in a 3-book z196 or z10 EC server. Table 10-5 Fanout AID numbers for System z10 EC Fanout location 160 Book Fourth First Third Second D1 00 08 10 18 D2 01 09 11 19 D3 N/A N/A N/A N/A D4 N/A N/A N/A N/A D5 02 0A 12 1A D6 03 0B 13 1B D7 04 0C 14 1C D8 05 0D 15 1D D9 06 0E 16 1E DA 07 0F 17 1F IBM System z Connectivity Handbook 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Important Note: The AID numbers shown in Table 10-5 are only valid for a new build server or for new books added. If a fanout is moved the AID will follow it to its new physical location. The AID assignment on a new build z10 BC for each fanout slot in relation to location is shown in Table 10-6. Table 10-6 AID number assignment for System z10 BC Fanout location AID D3 D4 D5 D6 D7 D8 00 01 02 03 04 05 Slots D1, D2, D9, and DA in the CPC drawer of a z10 BC have no fanout cards installed. Note: The AID numbers shown in Table 10-6 are only valid for a new build servers. If a fanout is moved the AID will follow it to its new physical location. The AID assigned to a fanout can be found in the PCHID report provided for each new server or for upgrades. Example 10-1 shows part of a PCHID report for a z10 EC model E26. In this example, two fanouts are installed in the first and second book, both in location D6. The assigned AID for the fanout in the first book (LG06) is 0B and the AID assigned to the fanout in the second book (LG15) is 1B. The port numbers are used by HCD/IOCDS for link definitions and are not shown here. Left to right, the ports on each fanout are numbered J01 and J02. Example 10-1 System z10 EC CHPID report CHPIDSTART 19756694 PCHID Machine: 2097-E26 SNxxxxxxx - - - - - - - - - - - - - - - - - Source Cage Slot F/C 06/D6 A25B D606 0163 15/D6 A25B D615 0163 REPORT - - - - - - - - - - - - - - - - - - - - PCHID/Ports or AID Comment AID=0B AID=1B For more information about System z InfiniBand support, refer to Getting Started with InfiniBand on System z10 and System z9, SG24-7539. Chapter 10. Coupling links and common time 161 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm 10.2.6 Fanout rebalancing (FC 2400) In a new build server with multiple books installed, the fanouts are balanced across the installed books following the plugging rules for new build servers. When adding books while upgrading, the existing fanouts remain in their original location and will not automatically be moved to the newly installed book. This may result in an unbalanced placement of fanouts, where one book may be fully populated with fanouts, and another (new) book has no fanouts installed. On a z9 EC: 򐂰 Fanouts used for I/O are rebalanced on a book-add. 򐂰 Fanouts used for PSIFB and ICBs are rebalanced with FC2400. On a z10 EC: 򐂰 Fanouts used for I/O and PSIFB are rebalanced on a book-add. 򐂰 Only fanouts used for ICBs are rebalanced with FC2400. On a z196: 򐂰 Fanouts used for I/O and PSIFB are rebalanced on a book-add. 򐂰 Since z196 does not support ICB, FC2400 is not applicable to z196. Important: Installation of this feature is disruptive. 10.3 Time functions There is a long-standing requirement for accurate time and date information in data processing. As single operating systems have been replaced by multiple, coupled operating systems on multiple servers, this need has evolved into a requirement for both accurate and consistent clocks among these systems. Clocks are said to be consistent when the difference or offset between them is sufficiently small. An accurate clock is consistent with a standard time source. The IBM z/Architecture, Server Time Protocol (STP) and External Time Reference (ETR) architecture facilitates the synchronization of server time-of-day (TOD) clocks to ensure consistent time stamp data across multiple servers and operating systems. The STP or ETR architecture provides a means of synchronizing TOD clocks in different servers with a centralized time reference, which in turn may be set accurately on the basis of an international time standard (External Time Source). The architecture defines a time-signal protocol and a distribution network, which permits accurate setting, maintenance, and consistency of TOD clocks. ETR attachments The ETR feature in a System z9 and System z10 servers provides the interface to either a Network Time Protocol (NTP) server with pulse per second (PPS) support (see 10.3.1, “NTP server with PPS support” on page 164) or an IBM Sysplex Timer (see 10.3.2, “Sysplex Timer” on page 165). The zEnterprise 196 does not support the IBM Sysplex Timer and ETR attachment. In an expanded availability configuration (two ETR features), each ETR card connects to either a different NTP server with PPS support or to a different Sysplex Timer. 162 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch11.fm STP facility Server Time Protocol (STP) is a server-wide facility that is implemented in the Licensed Internal Code of System z servers, and coupling facilities. STP presents a single view of time to PR/SM and provides the capability for multiple servers and coupling facilities to maintain time synchronization with each other. Any System z servers, or CFs may be enabled for STP by installing the STP feature. Each server and CF that are planned to be configured in a coordinated timing network (CTN) must be STP-enabled. Network Time Protocol (NTP) client support is available to the STP code on the z196, z10 and on z9. With this functionality the z196, z10 and z9 can be configured to use an NTP server as an external time source (ETS). NTP server with pulse per second (PPS) is supported. A z196 can not be connected to a Sysplex Timer®, we recommend migration to a STP-only Coordinated Time Network (CTN) for existing environments. It is possible to have a z196 as a Stratum 2 or Stratum 3 server in a Mixed CTN, as long as there are at least two System z10 or System z9 servers attached to the Sysplex Timer® operating as Stratum 1 servers. zEnterprise 196 ECF feature Two external clock facility (ECF) cards are a standard feature of the z196 server. The ECF cards provide a dual-path interface for Pulse Per Second (PPS) support. A cable connection from the PPS port on the ECF card to the PPS output of the NTP server is required when the z196 is using STP and configured in an STP-only CTN using NTP with pulse per second as the external time source. Each of the two standard ECF cards has one pulse per second (PPS) port for a coaxial cable connection to a PPS port on a Network Time Protocol (NTP) server. The coaxial cable must be terminated with a 50 ohm BNC connector. To sustain accuracy within 10 microseconds, the maximum coaxial cable length allowed is 50 feet. However, a fiber optic link can be used for extended distances up to 2 km. Note that a fiber optic converter kit is needed for such a configuration. System z10 ETR feature The External Time Reference (ETR) is a standard feature on z10 servers providing two ETR cards plugged in the z10 EC CEC cage or z10 BC CPC drawer. The ETR cards support concurrent maintenance. Each of the two standard ETR cards has one pulse per second (PPS) port for a coaxial cable connection to a PPS port on a Network Time Protocol (NTP) server. The coaxial cable must be terminated with a 50 ohm BNC connector. To sustain accuracy within 10 microseconds, the maximum coaxial cable length allowed is 50 feet. However, a fiber optic link can be used for extended distances up to 2 km. Note that a fiber optic converter kit is needed for such a configuration. Each feature also has a single port supporting an MT-RJ fiber optic connector to provide the capability to attach to a Sysplex Timer Unit. The Sysplex Timer Unit has an optical transceiver that supports an ESCON Duplex connector. Chapter 10. Coupling links and common time 163 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Note: The ETR card does not support a multimode fiber optic cable terminated with an ESCON Duplex connector as on the Sysplex Timer. Current 62.5 micron multimode ESCON Duplex jumper cables can be reused to connect to the ETR card. This is done by installing an MT-RJ/ESCON conversion kit between the ETR card MT-RJ port and the ESCON Duplex jumper cable. Fiber optic conversion kits and Mode Conditioning Patch (MCP) cables are not orderable as features on z10 servers. Fiber optic cables, cable planning, labeling, and installation are all customer responsibilities for new z10 installations and upgrades. IBM Fiber Cabling Services offer a total cable solution service to help with cable ordering needs, and are highly recommended. System z9 ETR feature The ETR feature (FC 6155) is optional on System z9 servers. Two ETR features can be installed in the CEC cage. The ETR card has one pulse per second (PPS) port for a coaxial cable connection to a PPS port on a Network Time Protocol (NTP) server. The coaxial cable must be terminated with a 50 ohm BNC connector. To sustain accuracy within 10 microseconds, the maximum coaxial cable length allowed is 50 feet. However, a fiber optic link can be used for extended distances up to 2 km. Note that a fiber optic converter kit is needed for such a configuration. Each feature also has a single port supporting an MT-RJ fiber optic connector that can be attached to a Sysplex Timer Unit. The Sysplex Timer Unit has an optical transceiver that supports an ESCON Duplex connector. Note: The System z9 ETR feature does not support a multimode fiber optic cable terminated with an ESCON Duplex connector. However, 62.5 micron multimode ESCON Duplex jumper cables can be reused to connect to the ETR feature. This is done by installing an MT-RJ/ESCON conversion kit between the ETR feature MT-RJ port and the ESCON Duplex jumper cable. An MT-RJ/ESCON conversion kit supplies a 62.5 micron multimode conversion cable. The conversion cable is two meters (6.5 feet) in length and is terminated at one end with an MT-RJ connector and at the opposite end with an ESCON Duplex receptacle to attach to the under floor cabling. See “Conversion kits” on page 200 for more information. Note: Fiber optic conversion kits and Mode Conditioning Patch (MCP) cables are not orderable as features on z10 and z9 servers. 10.3.1 NTP server with PPS support STP has the capability of configuring a Network Time Protocol (NTP) server that has a pulse per second (PPS) output signal as its External Time Source (ETS). This type of ETS device is available worldwide from several vendors that provide network timing solutions. The NTP output of the NTP server has to be connected to the Support Element (SE) LAN because the NTP client is running on the SE. Also the PPS output of the same NTP server has to be connected to the PPS input port on the External Clock Facility (ECF) card of the 164 IBM System z Connectivity Handbook 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm zEnterprise 196 server, or to the External Time Reference (ETR) card of the System z10 or System z9 server. STP tracks to the PPS signal to maintain time accuracy for the Coordinated Timing Network (CTN). STP is designed to maintain an accuracy of 10 microseconds to the PPS input signal. A number of variables, such as the cable used to connect the PPS signal and the accuracy of the NTP server to its time source (GPS or radio signals), will determine the ultimate accuracy of STP to Coordinated Universal Time (UTC), Figure 10-4 shows an expanded availability configuration with two ETR features in a System z10 server; each ETR card is connected to a different NTP server with PPS support. On a z196 server the PPS signal is connected to the External Clock Facility (ECF) card. NTP server Stratum 1 NTP server Stratum 1 July 14 14:21:00 2007 UTC July 14 14:21:00 2007 UTC PPS output PPS output Ethernet Switch System z HMC NTP client ETR card PPS port 0 ETR card PPS port 1 Figure 10-4 NTP server with PPS: expanded availability configuration For more information about NTP support, refer to: http://www-03.ibm.com/systems/z/pso/stp.html 10.3.2 Sysplex Timer The Sysplex Timer provides the synchronization for the time-of-day clocks of multiple servers, and thereby allows events started by different servers to be properly sequenced in time. When multiple servers update the same database, all updates are required to be time stamped in proper sequence. More information about the IBM Sysplex Timer can be found in S/390 Time Management and IBM Sysplex Timer, SG24-2070. Chapter 10. Coupling links and common time 165 5444ch11.fm Draft Document for Review July 21, 2010 6:06 pm Note: IBM Sysplex Timer Model 1 and IBM Sysplex Timer Model 2 have both been withdrawn from marketing. Service support for the IBM Sysplex Timer Model 1 is discontinued. All strategic planning should include a plan to implement or migrate to a Coordinated Timing Network, using the Server Time Protocol Facility and NTP support. IBM Sysplex Timer is not supported on the z196. 10.3.3 Software support Software requirements vary with the actual design and usage of common time. 򐂰 All in-service z/OS releases support ETR. 򐂰 z/OS V1R7 plus PTFs (or later) is required for STP. For more information regarding STP software support, go to: http://www-03.ibm.com/systems/z/pso/stp/software.html 10.4 References To help you understand, plan, and implement a Parallel Sysplex, go to: http://www-1.ibm.com/servers/eserver/zseries/pso The following additional IBM publications provide detailed information about Parallel Sysplex: 򐂰 Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367 򐂰 Sysplex Timer Planning, GA23-0365 򐂰 Planning for the 9037 Model 2, SA22-7233 򐂰 Server Time Protocol Planning Guide, SG24-7280 򐂰 Server Time Protocol Implementation Guide, SG24-7281 򐂰 Getting the Most Out of a Parallel Sysplex, SG24-2073 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 򐂰 IBM System z10 Enterprise Class Technical Introduction, SG24-7515 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 Business Class Technical Introduction, SG24-7241 166 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch12.fm 11 Chapter 11. Extended distance solutions This chapter describes architectural requirements and implementation solutions for System z connectivity over extended distances. The following distance solution topics are covered: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Fiber optic link unrepeated distances Enterprise Systems Connection (ESCON) Fibre Connection (FICON) InterSystem Channel (ISC) Sysplex Timer links (ETR and CLO) Server Time Protocol (STP) GDPS WDM technologies System z qualified WDM vendors Note: Dark fiber is required for any System z extended distance solution that involves multiple site connectivity. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 167 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm 11.1 Unrepeated distances This section lists the maximum unrepeated distance and link budget for each type of System z fiber optic link. Longer distances are possible using repeaters, switches, channel extenders, and Wavelength Division Multiplexing (WDM). In Table 11-1, a link is a physical connection over a transmission medium (fiber) used between an optical transmitter and an optical receiver. The maximum allowable link loss, or link budget, is the maximum amount of link attenuation (loss of light), expressed in decibels (dB), that can occur without causing a possible failure condition (bit errors). When using multimode fiber, as the link data rate increases the unrepeated distance and link budget decreases. The link budget is derived from combining the channel insertion loss budget with the unallocated link margin budget. The link budget numbers have been rounded to the nearest tenth of a dB. Table 11-1 Fiber optic connections: unrepeated distances Feature type Fiber type Link data rate Maximum distance Link budget MM 62.5 µm 200 Mbps 500 800 2 km 3 km 8 dB MM 50 µm 200 Mbps 800 2 km 8 dB MM 62.5 µm 8 Mbps Any 3 km 26 km 8 dB MM 50 µm 8 Mbps Any 2 km 4 km 8 dB SM 9 µm 2 Gbps N/A 10 km 12 km 7.8 dB MM 62.5 µm 2 Gbps 200 120 m 2.8 dB MM 50 µm 2 Gbps 500 300 m 2.2 dB FICON 4KM LX SM 9 µm 4 Gbps N/A 4 km 4.8 dB FICON 10KM LX SM 9 µm 4 Gbps N/A 10 km 7.8 dB MM 62.5 µm 4 Gbps 160 200 55 m 70 m 2.1 dB 2.0 dB MM 50 µm 4 Gbps 500 2000 150 m 270 m 2.3 dB 2.5 dB ESCON (SBCON) ETR (for Sysplex Timer) FICON LX FICON SX FICON SX 168 Fiber B/W MHz-km PSIFB LR (HCA2-O LR): 1x IB-SDR 1x IB-DDR SM 9 µm PSIFB (HCA2-O): 12 x IB-SDR 12 x IB-DDR MM 50 µma ISC-3 peer mode at 2 Gbps SM 9 µm IBM System z Connectivity Handbook N/A 10 km 5.7 dB 2000 150 m 2.1 dB N/A 10 km 12 kmb 7 dB 2.5 Gbps 5 Gbps 3 GBps 6 GBps 2 Gbps 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm Feature type Fiber type ISC-3 peer mode at 1 Gbpsb SM 9 µm 1 Gbps SM 9 µm Gigabit Ethernet LX Link data rate Maximum distance Link budget N/A 10 km 20 km 7 dB 1 Gbps N/A 5 km 4.6 dB MM 62.5 µmc 1 Gbps 500 550 m 2.4 dB MM 50 µmc 1 Gbps 500 550 m 2.4 dB MM 62.5 µm 1 Gbps 166 200 220 m 275 m 2.6 dB MM 50 µm 1 Gbps 500 550 m 3.6 dB SM 9 µm 10 Gbps N/A 10 km 6 dB MM 62.5 µm 10 Gbps 200 33 m 2.5 dB MM 50 µm 10 Gbps 2000 500 300 m 82 m 2.6 dB 2.3 dB Gigabit Ethernet SX 10 Gigabit Ethernet LR 10 Gigabit Ethernet SR Fiber B/W MHz-km a. Requires OM3 fiber optic cable with MPO connectors. b. Requires RPQ 8P2197. This RPQ provides an ISC-3 Daughter Card which clocks at 1 Gbps in peer mode. This allows the ISC-3 peer mode link to have an unrepeated distance extended up to 20 km. Under certain conditions, RPQ 8P2263 (or RPQ 8P2340 for the z10 BC) may be required in conjunction with RPQ 8P2197. Check with your IBM representative. c. Requires fiber optic Mode Conditioning Patch (MCP) cables. Notes for Table 11-1 on page 168: 򐂰 SBCON is the non-IBM trademarked name of the ANSI industry standard for ESCON. 򐂰 All industry-standard links (ESCON/SBCON, FICON, Gigabit Ethernet) follow published industry standards. The minimum fiber bandwidth requirement to achieve the distances listed is applicable for multimode (MM) fiber only. There is no minimum bandwidth requirement for single mode (SM) fiber. 򐂰 The bit rates given may not correspond to the effective channel data rate in a given application due to protocol overheads and other factors. 򐂰 LC Duplex and SC Duplex connectors are keyed per the ANSI Fibre Channel Standard specifications. 򐂰 MCP denotes mode conditioning patch cable, which is required to operate some links over multimode fiber. 򐂰 The 16-port ESCON feature supports an MT-RJ connector. A conversion kit (MM 62.5 ESCON Duplex receptacle to MT-RJ connector) can be used to utilize existing multimode ESCON Duplex cable infrastructure. 򐂰 The ISC-3 feature supports an LC Duplex connector. A conversion kit (SM SC Duplex receptacle to LC Duplex connector) can be used to utilize existing single mode SC Duplex cable infrastructure. 򐂰 The FICON Express2 and FICON Express features allow an auto-negotiated link speed of either 1 Gbps or 2 Gbps. The specifications in the table are for 2 Gbps link rate. MCP Cables are not supported for 2 Gbps links. 򐂰 The FICON Express4 features allow an auto-negotiated link speed of either 1 Gbps, 2 Gbps, or 4 Gbps. MCP Cables are not supported for 2 Gbps or 4 Gbps links. Chapter 11. Extended distance solutions 169 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 The FICON Express8 features allow an auto-negotiated link speed of either 2 Gbps, 4 Gbps, or 8 Gbps. MCP Cables are not supported with FICON Express8 features. 򐂰 As light signals traverse a fiber optic cable, the signal loses some of its strength (decibels (dB) is the metric used to measure light power loss). The significant factors that contribute to light power loss are the length of the fiber, the number of splices, and the number of connections. The amount of light power loss (dB) across a link is known as the Link Budget. All links are rated for a maximum Link Budget (the sum of the applicable light power loss factors must be less than the Link Budget) and a maximum distance (exceeding the maximum distance will cause undetectable data integrity exposures). Another factor that limits distance is jitter, but this is typically not a problem at these distances. 򐂰 Link Budget and fiber bandwidth should be measured at the appropriate wavelength: long wavelength (1300 nm) or short wavelength (850 nm). For planning purposes, the following worst case values can be used to estimate the Link Budget. Refer to the references listed and contact the fiber vendor for specific values, which may be different for your configuration: Link loss at 1300 nm = 0.50 db/km Link loss per splice = 0.15 db/splice (not dependent on wavelength) Link loss per connection = 0.50 db/connection (not dependent on wavelength) 򐂰 Deviations from these specifications (longer distance or Link Budget) may be possible. They are evaluated on an individual basis by submitting a Request for Price Quote (RPQ) to IBM. Note: For extended distances also see 11.6, “Wavelength Division Multiplexing (WDM)” on page 181. 11.2 Conversion to parallel channel The z10 and z9 servers do not support direct connection of parallel channels. Therefore, an ESCON to parallel channel converter is required. Parallel channel repeated distance solutions One connectivity solution for parallel channel attached I/O control units and devices over extended distances is ESCON conversion to parallel channel. ESCON conversion to parallel channel ESCON to parallel channel converters can be used for: 򐂰 Connecting parallel channel attached I/O control units and devices over extended distances. 򐂰 Connecting parallel channel attached I/O control units and devices to System z servers. The IBM 9034 ESCON Converter (ESCC) model 1, and the 34600 FXBT ESCON Converter from Optica Technologies Incorporated, allow parallel channel attached I/O control units to connect to an ESCON channel. Both products convert the ESCON channel’s optical signals to parallel channel electrical signals, and the control unit’s parallel channel electrical signals to ESCON channel optical signals. System z servers support ESCON converter connection via the ESCON conversion channel CHPID types CVC and CBY. Note: The IBM 9034 ESCON Converter model 1 is withdrawn from marketing. 170 IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm The 34600 FXBT ESCON converter is functionally compatible with the IBM 9034. For more information, including supported parallel channel I/O control units and devices, see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by other vendors. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. Each ESCON converter, which attaches one channel to up to eight control units in the same multidrop (daisy chained) configuration, can be located (Figure 11-1): 򐂰 Between an ESCON channel and a single control unit. 򐂰 Between an ESCON channel and the first control unit in a multidrop configuration. 򐂰 Between an ESCON Director port and a control unit. A two-port dedicated (static) connection is required within the ESCON Director when attaching to an ESCON conversion (CVC) channel. ESCON conversion channel ESCON link ESCON converter Parallel 3 km with 62.5 µm fiber 122 m 2 km with 50 µm fiber 400 feet Parallel CU Device Parallel CU Device Parallel CU Device Parallel CU Device Parallel CU Device Parallel CU Device ESCON conversion channel with ESCON Director ESCON Director ESCON link ESCON link ESCON converter Parallel static port connection Up to: 3 km with 62.5 µm fiber Up to: 3 km with 62.5 µm fiber 2 km with 50 µm fiber 2 km with 50 µm fiber Total cannot exceed: 3 km with 62.5 µm fiber 122 m 2 km with 50 µm fiber 400 feet Figure 11-1 ESCON to parallel channel connection The distance between the ESCON converter and the control unit cannot exceed the maximum distances supported for the parallel channel I/O environment. Special limitations can exist, depending on the control units or devices used. The maximum channel data rate supported by an ESCON converter is 4.5 MBps, but the actual data rate depends on the attached control unit. Chapter 11. Extended distance solutions 171 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm The maximum supported distance of the entire ESCON path (with or without an ESCON Director) connecting an ESCON channel to an ESCON converter, using multimode fiber optic cables, is: 򐂰 3 km (1.86 miles) when using 62.5 micron (µm) fiber 򐂰 2 km (1.24 miles) when using 50 micron (µm) fiber Note: The ESCON path connecting to an ESCON converter cannot be extended beyond the maximum supported distances using other technologies, such as Wavelength Division Multiplexing (WDM) or ESCON Director XDF feature. The maximum supported distance of the entire ESCON path connecting to an ESCON converter is further restricted by the type of attached control unit. For example, when using 62.5 micron (µm) fiber, an ESCON converter attached to an IBM 3420 model 8 tape drive and IBM 3803 model 2 tape control unit supports: 򐂰 A maximum ESCON link distance of 2.8 km (1.74 miles), without an ESCON Director 򐂰 A maximum entire ESCON path (combined ESCON link) distance of 2.6 km (1.61 miles), with an ESCON Director 11.3 Enterprise Systems Connection (ESCON) This section describes architectural requirements and implementation solutions for ESCON channel connectivity over unrepeated and repeated distances. The z10 and z9 servers support one type of ESCON channel feature: the 16-port ESCON feature (feature code 2323). For more information about this feature, see 4.2.1, “ESCON channel support on System z” on page 44. 11.3.1 ESCON unrepeated distance The maximum unrepeated distance of an ESCON link using multimode fiber optic cables is (Figure 11-2): 򐂰 3 km (1.86 miles) when using 62.5 micron (µm) fiber 򐂰 2 km (1.24 miles) when using 50 micron (µm) fiber Point-to-point ESCON link ESCON CU 3 km with 62.5 µm fiber 2 km with 50 µm fiber Figure 11-2 ESCON unrepeated distance 11.3.2 ESCON repeated distance solutions This section describes a number of extended distance connectivity solutions for ESCON channel attached I/O control units and devices. 172 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ch12.fm The repeated distance for a native mode ESCON channel has a System z qualified design point of 100 km (62 miles). The actual maximum end-to-end distance is limited to much less than this by the specific characteristics of the attached control units and devices. Many ESCON control units and devices support a maximum repeated distance of 43 km (26.72 miles). Others are limited to 26 km (16.16 miles) or less. The supported distance is control unit specific, therefore you must refer to each control unit’s characteristics manual or specification manual. ESCON data rate droop It is important to remember that the effective channel data rate of an ESCON channel is affected by distance. Each channel command sent from the ESCON channel to the control unit has to complete before the channel will send the next command. This requires an exchange of interlocking protocols for each command, and there are a number of these protocol exchanges during the transfer of the command and data for one Channel Control Word (CCW). This hand shaking protocol increases the elapsed time for each CCW, and has a significant effect on data rate droop at greater channel to control unit distances. Droop is a matter of physics. Droop begins when the link distance reaches the point where the time light takes to make one round trip on the link is equal to the time it takes to transmit the number of bytes that will fit in the receiver’s buffer. Therefore, as the distance between an ESCON channel and a control unit increases, the effective channel data rate decreases, with significant data rate droop occurring at distances beyond 9 km (5.6 miles). At 20 km (12.43 miles), an ESCON channel delivers less than half the maximum effective data rate it provides at 0 km (co-located server and control unit). In all cases, the basic speed of light propagation delay through fiber (5 microseconds (µs) per kilometer (km) one way, or 10 µs per km round trip) will add 0.1 milliseconds (ms) of increased response time for every 10 km of fiber distance. Notes: The actual maximum end-to-end distance is limited by the specific characteristics of the attached control units and devices. Peer to Peer Remote Copy ESCON (PPRC ESCON) has a maximum supported distance of 103 km (64 miles). PPRC ESCON exploits ESCON link level hardware (ESCON CU ports, ESCON fiber optic cables, and connectors), plus ESCON protocol data frames, as the connectivity between IBM TotalStorage® Enterprise Storage Server® (ESS) primary volumes and ESS secondary volumes. However, PPRC uses a different protocol handshake architecture compared with native ESCON channel. This significantly reduces the effects of data rate droop, as experienced by native ESCON channels, allowing a far greater data rate at extended distances. Chapter 11. Extended distance solutions 173 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm Channel path with one ESCON Director One ESCON Director increases the maximum supported distance of the ESCON channel path (Figure 11-3) to: 򐂰 6 km (3.73 miles) when using 62.5 micron (µm) fiber 򐂰 4 km (2.48 miles) when using 50 micron (µm) fiber Switched point-to-point ESCON CU ESCON Director ESCON link ESCON links ESCON CU ESCON CU 3 km with 62.5 µm fiber 3 km with 62.5 µm fiber 2 km with 50 µm fiber 2 km with 50 µm fiber Figure 11-3 ESCON path with one Director Channel path with chained ESCON Directors Chained ESCON Directors increase the maximum supported distance of the ESCON channel path (Figure 11-4) to: 򐂰 9 km (5.59 miles) when using 62.5 micron (µm) fiber 򐂰 6 km (3.73 miles) when using 50 micron (µm) fiber Channel path with chained ESCON Directors with XDF feature The IBM 9032 Model 5 eXtended Distance Feature (XDF) feature allows the link between two chained ESCON Directors to be extended up to 20 km (12.43 miles), with 9 micron (µm) single mode fiber optic cables. This increases the maximum supported distance of the ESCON channel path (Figure 11-4) to: 򐂰 26 km (16.16 miles) when using 62.5 micron (µm) fiber 򐂰 24 km (14.91 miles) when using 50 micron (µm) fiber Chained ESCON Directors ESCON Director ESCON link ESCON CU ESCON Director ESCON link ESCON links static port connection ESCON CU ESCON CU 20 km with 9 µm fiber and XDF 3 km with 62.5 µm fiber 3 km with 62.5 µm fiber 3 km with 62.5 µm fiber 2 km with 50 µm fiber 2 km with 50 µm fiber 2 km with 50 µm fiber Figure 11-4 ESCON path with chained Directors Channel path with one ESCON Director with FICON Bridge port feature The FICON Bridge port feature provides FICON to ESCON conversion (CHPID type FCV) with simultaneous connections to 8 different control units on 8 different ESCON links. The FICON LX Bridge port feature only supports FICON Express LX links at a link data rate of 1 Gbps. It does not support FICON Express LX links operating at 2 Gbps, or FICON Express SX. 174 IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm FICON Express2 does not support FICON to ESCON conversion (CHPID type FCV) channels or FICON Bridge. Note: FICON Bridge is an aid for ESCON to FICON migration only, not a long term connectivity solution. The FICON conversion link, between the System z server FICON channel port and ESCON Director FICON Bridge port, has an unrepeated distance of 10 km (6.21 miles) with 9 micron (µm) single mode fiber optic cables. This distance can be extended up to 20 km (12.43 miles) with RPQ 8P2263 (System z Extended Distance) or RPQ 8P2340 for the z10 BC. This increases the maximum supported distance of the entire channel path (Figure 11-5) to: 򐂰 13 km (8.08 miles) when using 62.5 micron (µm) fiber (23 km (14.29 miles) with RPQ) 򐂰 12 km (7.46 miles) when using 50 micron (µm) fiber (22 km (13.67 miles) with RPQ) ESCON Director with FICON Bridge feature ESCON CU ESCON Director FICON conversion link FICON LX 1Gbps FCV mode ESCON links FICON Bridge feature ESCON CU ESCON CU 10 km with 9 µm fiber 3 km with 62.5 µm fiber 20 km with 9 µm fiber and RPQ 8P2263 2 km with 50 µm fiber Figure 11-5 ESCON path with one Director with FICON Bridge Channel path with chained Directors and FICON Bridge port feature When an ESCON Director with a FICON Bridge feature installed is in a chained ESCON Director configuration, it must be the first Director in the channel path. The second ESCON Director in the channel path must have a static connection defined. This increases the maximum supported distance of the entire channel path (Figure 11-6) to: 򐂰 16 km (9.94 miles) when using 62.5 micron (µm) fiber (26 km (16.16 miles) with RPQ) 򐂰 14 km (8.70 miles) when using 50 micron (µm) fiber (24 km (14.91 miles) with RPQ) Chained ESCON Directors with FICON Bridge ESCON Director ESCON Director FICON conversion link ESCON link ESCON link ESCON CU static port connection FICON Bridge feature 10 km with 9 µm fiber 3 km with 62.5 µm fiber 3 km with 62.5 µm fiber 20 km with 9 µm fiber / RPQ 8P2263 2 km with 50 µm fiber 2 km with 50 µm fiber Figure 11-6 ESCON path with chained Directors and FICON Bridge Important: The IBM ESCON Director 9032 (including all models and features) is withdrawn from marketing. There is no IBM replacement for the IBM 9032. Chapter 11. Extended distance solutions 175 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm The PRIZM Protocol Converter Appliance from Optica Technologies Incorporated provides a FICON to ESCON conversion function that has been System z qualified. Qualification letters can be found on the IBM System z I/O Connectivity Web page: http://www-03.ibm.com/systems/z/hardware/connectivity/products/index.html Select the Products tab, then FICON / FCP Connectivity. Scroll down to the “other supported devices” area on the Web page. For more information about the PRIZM Protocol Converter Appliance, see: http://www.opticatech.com/ Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims by vendors for products that have not been System z qualified. Questions regarding these capabilities and device support should be addressed to the suppliers of those products. WDM technologies There are other extended distance connectivity technologies available to extend ESCON and other link types, for example Wavelength Division Multiplexer (WDM). WDM technology also provides increased flexibility in that multiple links and protocol types can be transported over a single dark fiber trunk. See 11.6.2, “System z qualified WDM vendor products” on page 183. 11.4 Fibre Connection (FICON) This section describes architectural requirements and implementation solutions for FICON channel connectivity over unrepeated and repeated distances. The term FICON represents the architecture as defined by the InterNational Committee of Information Technology Standards (INCITS), and published as ANSI standards. FICON also represents the names of the System z server feature types: FICON Express4, FICON Express2, and FICON Express. All feature types support a long wavelength (LX) laser version and a short wavelength (SX) laser version. They support native FICON (FC, FCTC), and Fibre Channel Protocol (FCP) channel modes. The FICON Express LX feature supports FICON Conversion (FCV) channel mode. FICON Express8, FICON Express4 and FICON Express2 features do not support FCV mode. For more information about these features see 5.3, “Connectivity” on page 73. 11.4.1 FICON unrepeated distance The unrepeated distance supported by System z FICON features is dependent upon: 򐂰 The feature port transceiver type (LX or SX). 򐂰 The fiber type being used: 9 micron (µm) single mode, or 50 micron (µm) or 62.5 micron (µm) multimode. Also, for multimode, the fiber bandwidth (MHz.km) of the fiber. 򐂰 The speed at which the feature port is operating. 򐂰 Whether there are Mode Conditioning Patch (MCP) cables in the fiber optic link. 176 IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm Cabling specifications are defined by the Fibre Channel - Physical Interface - 4 (FC-PI-4) standard and used by System z FICON features. Table 11-2 identifies cabling types and link data rates that are supported in the FICON environment, including their allowable maximum distances and link loss budget. The link loss budget is derived from the channel insertion loss budget defined by the FC-PI-4 standard (Revision 8.00). Table 11-2 Fiber optic cabling for FICON: maximum distances and link loss budget FC-PI-4 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps ISLa Fiber core (light source) Distance in meters (in feet) Link loss budget in dB Distance in meters (in feet) Link loss budget in dB Distance in meters (in feet) Link loss budget in dB Distance in meters (in feet) Link loss budget in dB Distance in meters (in feet) Link loss budget in dB 9 µm SM (10 km LX laser) 10000 (32736) 7.8 10000 (32736) 7.8 10000 (32736) 7.8 10000 (32736) 6.4 10000 (32736) 6.0 9 µm SM ( 4 km LX laser) 4000 (13200) 4.8 4000 (13200) 4.8 4000 (13200) 4.8 NA NA NA NA 50 µm MMb (SX laser) 860 (2822) 4.62 500 (1640) 3.31 380 (1247) 2.88 150 (492) 2.04 300 (984) 2.6 50 µm MMc (SX laser) 500 (1640) 3.85 300 (984) 2.62 150 (492) 2.06 50 (164) 1.68 82 (269) 2.3 62.5 µm MMd (SX laser) 300 (984) 3.0 150 (492) 2.1 70 (230) 1.78 21 (69) 1.58 33 (108) 2.4 a. Inter-Switch Link (ISL) between two FICON Directors. b. OM3: 50/125 µm laser optimized multimode fiber with a minimum overfilled launch bandwidth of 1500 MHz-km at 850nm as well as an effective laser launch bandwidth of 2000 MHz-km at 850 nm in accordance with IEC 60793-2-10 Type A1a.2 fiber. c. OM2: 50/125 µm multimode fiber with a bandwidth of 500 MHzkm at 850 nm and 500 MHz-km at 1300 nm in accordance with IEC 60793-2-10 Type A1a.1 fiber. d. OM1: 62.5/125 µm multimode fiber with a minimum overfilled launch bandwidth of 200 MHzkm at 850 nm and 500 MHz-km at 1300 nm in accordance with IEC 60793-2-10 Type A1b fiber. Note: IBM does not support a mix of 50 µm and 62.5 µm fiber optic cabling in the same physical link. For more information see Table 11-1 on page 168 and Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367. 11.4.2 FICON repeated distance solutions This section describes a number of extended distance connectivity solutions for FICON channel attached I/O control units and devices. The repeated distance for a FICON channel is System z qualified to a maximum of 100 km (62 miles). For all FICON features using repeaters, the end-to-end distance between the FICON channel and the FICON Director port can be up to 100 km. The same end-to-end distance is also available between the FICON Director port and the control unit port. However, the overall end-to-end distance between the FICON channel and control unit is System z qualified for up to 100 km only. Chapter 11. Extended distance solutions 177 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm FICON data rate droop FICON channels (CHPID type FC, FCP, and FCV) are far superior to ESCON channels at increased distances. In particular, ESCON channels experience higher response time elongations than FICON as distances increase, and an ESCON channel experiences a much more significant data rate droop at much shorter distances than FICON channels. There are two main reasons for this difference between ESCON and FICON channels: 򐂰 Increased buffer sizes in the FICON channel feature card 򐂰 Improved protocols with fewer round trip handshakes across the link For ESCON links, significant data rate droop occurs at extended distances over 9 km. For FICON channels, the channel to control unit end-to-end distance can be increased up to 100 km without data rate performance droop occurring, when the FICON Director buffer credits are set appropriately. The number of buffer credits required will depend on the link data rate and the maximum number of buffer credits supported by the FICON Director or Control Unit, as well as application and workload characteristics. While it is theoretically possible for FICON to maintain high bandwidth at distances greater than 100 km, these distances have not been System z qualified, and are only achievable if enough buffer credits exist to support the link speed. The distance capability with FICON is becoming increasingly important as movement occurs towards remote I/O, vaulting for disaster recovery, business continuity, and GDPS for availability. FICON LX channel path with one FICON Director One FICON Director increases the maximum supported distance of a FICON LX channel path using single mode fiber optic cables (Figure 11-7) to: 򐂰 14 km (8.69 miles) for 1 Gbps, 2 Gbps, and 4 Gbps LX links with FICON Express4 LX 4KM features 򐂰 20 km (12.43 miles) for 1 Gbps, 2 Gbps, and 4 Gbps LX links 򐂰 30 km (18.64 miles) for 1 Gbps LX links with RPQ 8P2263 or RPQ 8P2340 for the z10 BC 򐂰 22 km (13.67 miles) for 2 Gbps LX links with RPQ 8P2263 or RPQ 8P2340 for the z10 BC Packet switched point-to-point FICON Director FICON CU FICON link FICON links 9 µm fiber 9 µm fiber FICON LX FC or FCP mode LX 1 Gbps: 10 km / 20 km (RPQ 8P2263) FICON CU FICON CU 10 km LX 2 Gbps: 10 km / 12 km (RPQ 8P2263) LX 4 Gbps: 4 km / 10 km Figure 11-7 FICON LX path with one Director FICON LX channel path with two cascaded FICON Directors A FICON channel path can include a maximum of two cascaded FICON Directors. The maximum supported distance of the FICON channel path is FICON Director vendor specific. The use of extended distance long wavelength transceivers on the Inter-Switch links (ISL) between the FICON Directors may be required to achieve vendor quoted distances. Each ISL requires one fiber trunk (two fibers) between the FICON Directors. 178 IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm For example, using the configuration illustrated in Figure 11-8 and assuming the distance between the two FICON Directors is 10 km, the maximum supported distance of a FICON LX channel path using single mode fiber optic cables would be: 򐂰 30 km (18.64 miles) for 1 Gbps, 2 Gbps, and 4 Gbps LX links 򐂰 40 km (24.86 miles) for 1 Gbps LX links with RPQ 8P2263 or RPQ 8P2340 for the z10 BC 򐂰 32 km (19.88 miles) for 2 Gbps LX links with RPQ 8P2263 or RPQ 8P2340 for the z10 BC 򐂰 24 km (14.9 miles) with FICON Express4 LX 4KM features for 1 Gbps, 2 Gbps, and 4 Gbps LX links Cascaded FICON Directors FICON Director FICON Director FICON CU FICON link FICON link FICON links 9 µm fiber 9 µm fiber 9 µm fiber FICON LX FC or FCP mode LX 1 Gbps: 10 km / 20 km (RPQ) FICON CU FICON CU distance is vendor specific 10 km LX 2 Gbps: 10 km / 12 km (RPQ) LX 4 Gbps: 4 km / 10 km Figure 11-8 FICON LX path with cascaded Directors FICON channel path: transceiver intermix A FICON channel path through one or two FICON Directors consists of multiple optical fiber links. Each link in the channel path can be either long wavelength (LX) or short wavelength (SX), allowing the channel path to be made up of an intermix of link types. This is possible because the FICON Director performs an optical to electrical to optical conversion (OEO) of the channel path as it passes through the Director. Note: The transceiver type (LX or SX) at each end of a given link must match. WDM technologies There are other extended distance connectivity technologies available to extend FICON and other link types, for example, Wavelength Division Multiplexer (WDM). WDM technology also provides increased flexibility in that multiple links and protocol types can be transported over a single dark fiber trunk. See 11.6, “Wavelength Division Multiplexing (WDM)” on page 181. 11.5 Coupling links This section describes architectural requirements and implementation solutions for Coupling link connectivity over unrepeated and repeated distances. Chapter 11. Extended distance solutions 179 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm Coupling link unrepeated distance Table 11-3 shows the maximum unrepeated distances and link data rates supported for Coupling links on System z servers. For more information see Table 11-1 on page 168 and the accompanying notes and Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367. Table 11-3 Coupling link unrepeated distance and link data rate support Coupling link type Maximum unrepeated distance Link data rate IC (internal) internal memory-to-memory (faster than ICB-4) ISC-3 peer mode at 2 Gbps 10 km 12 kma 2 Gbps ISC-3 peer mode at 1 Gbpsb 10km 20 kma 1 Gbps ICB-4 7/10 metersc 2 GBps ICB-3 7/10 metersc 1 GBps PSIFB 150 meters 6 GBps or 3 GBps PSIFB LR 10 km 2.5 Gbps or 5 Gbps a. Requires RPQ 8P2263 (System z Extended Distance) or RPQ 8P2340 for the z10 BC. b. Requires RPQ 8P2197. This RPQ provides an ISC-3 Daughter Card which clocks at 1 Gbps in peer mode. This allows the ISC-3 peer mode link to have an unrepeated distance extended up to 20 km. Under certain conditions, RPQ 8P2263 (or RPQ 8P2340 for the z10 BC) may be required in conjunction with RPQ 8P2197. Check with your IBM representative. c. ICB-4 and ICB-3 each use unique 10 m copper cables. The maximum distance between each server’s raised-floor entry cutout is 7 m due to cable routing. 180 IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm 11.6 Wavelength Division Multiplexing (WDM) WDM is a technique used to transmit several independent bit streams over a single fiber optic link; see Figure 11-9. It is an approach to opening up the conventional optical fiber bandwidth by breaking it up into many channels, each at a different optical wavelength (a different color of light). Each wavelength can carry a signal at any bit rate less than an upper limit defined by the electronics, typically up to several gigabits per second. ESCON Combining Signals Different light frequencies over a single fiber path Separating Signals ESCON FICON FICON ISC ISC GbE GbE Receivers Transmitters Figure 11-9 WDM transmission technique The channels are protocol-independent, so a wide variety of protocols is supported; for example, ESCON, FICON, FCP, InterSystem Channel (ISC), Sysplex Timer (CLO and ETR), Server Time Protocol (STP), and Gigabit Ethernet. Because the actual signal bandwidth that the electronics can handle over one wavelength is such a small fraction of the inter-channel spacing, the signals do not interfere with one another and can therefore be multiplexed into a single fiber using a passive grating multiplexer. Some extended distance uses of WDM technology are: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 ESCON and FICON/FCP channel connections to remote control units and devices LAN and network protocol connections to remote sites IBM System Storage Metro Mirror (synchronous Peer-to-Peer Remote Copy (PPRC)) IBM System Storage Global Mirror IBM System Storage z/OS Global Mirror (asynchronous Extended Remote Copy (XRC)) Peer-to-Peer Virtual Tape Server (PtP VTS), a form of remote copying tape data 11.6.1 System z GDPS qualification GDPS is an enterprise-wide continuous availability and disaster recovery automation solution that can manage recovery from planned and unplanned outages across distributed servers as well as System z servers. GDPS can be configured in either a single site or in a multisite configuration. It is designed to manage remote copy configuration between storage subsystems, automate Parallel Sysplex operational tasks, and perform failure recovery from a single point of control, thereby improving application availability. Historically, this solution was known as a Geographically Dispersed Parallel Sysplex™. Today, GDPS continues to be applied as a general term for a suite of business continuity solutions, including those that do not require a dispersed or multisite sysplex environment. Chapter 11. Extended distance solutions 181 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm GDPS supports the following forms of remote copy in multisite solutions: 򐂰 IBM System Storage Metro Mirror, synchronous Peer-to-Peer Remote Copy (PPRC) 򐂰 IBM System Storage Global Mirror, asynchronous Peer-to-Peer Remote Copy 򐂰 IBM System Storage z/OS Global Mirror, asynchronous Extended Remote Copy (XRC) Depending on the form of remote copy implemented, the solution is referred to as: 򐂰 GDPS/Metro Mirror (PPRC) 򐂰 GDPS/Global Mirror (PPRC) 򐂰 GDPS/z/OS Global Mirror (XRC) The GDPS solution is also independent of disk vendor, as long as the vendor meets the specific levels of Metro Mirror, Global Mirror, and z/OS Global Mirror architectures. For more information about GDPS, visit the GDPS Web site: http://www.ibm.com/systems/z/gdps/ IBM only supports WDM products that are qualified by System z for use in GDPS solutions. To obtain this qualification, WDM vendors obtain licensed IBM patents, intellectual property, and know-how that are related to the GDPS architecture. This gives vendors access to the proprietary IBM protocols and applications used in a GDPS environment [including Sysplex Timer, InterSystem Channel (ISC), Server Time Protocol (STP), Metro Mirror, Global Mirror, and z/OS Global Mirror. Licensing of IBM patents also provides the WDM vendor with technical information pertaining to future IBM releases. Qualified vendors will typically license this information for an extended period, allowing them to subscribe to the latest GDPS architecture changes and to be among the first to market with offerings that support these features. Note: We recommend that you check with your WDM vendor for current licensing status. In addition, these vendor products have been tested and qualified by IBM with the same test environment and procedures used to test the protocols that provide the required connectivity of a GDPS configuration. This testing includes functionality, recovery, and in some cases performance measurements. Having access to these test facilities allows IBM to configure a fully functional sysplex and to simulate failure and recovery actions that could not be tested as part of a working customer environment. IBM has the facilities to test and qualify these products with both current and previous generation equipment within the IBM Vendor Solutions Connectivity (VSC) Lab in Poughkeepsie, NY, U. S. This qualification testing allows IBM to reproduce any concerns that might arise while using this equipment in a customer’s application. Components The following GDPS components are used during the qualification process: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 182 IBM Parallel Sysplex IBM System Storage Optical Wavelength Division Multiplexer (WDM) IBM System Storage Metro Mirror (PPRC), a synchronous form of remote copy IBM System Storage Global Mirror IBM System Storage z/OS Global Mirror (XRC), an asynchronous form of remote copy IBM System z Connectivity Handbook 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm Protocols The following GDPS connectivity protocols are tested during the qualification process: 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 Enterprise Systems Connection (ESCON) IBM Sysplex Timer (ETR/CLO) Fibre Connection (FICON) Fibre Channel (FCP) Fibre Channel InterSwitch Links (ISL) InterSystem Channel-3 (ISC-3) peer mode (2 Gbps) InterSystem Channel-3 (ISC-3) peer mode (1 Gbps), via RPQ 8P2197 Server Time Protocol (STP) Often, these tested protocols are used in non-GDPS environments as well. The robust testing performed during the qualification process should provide customers with a high level of confidence when using these System z qualified optical WDM vendor products in non-GDPS environments. 11.6.2 System z qualified WDM vendor products The latest list of qualified WDM vendor products can be found through the Resource Link at: https://www.ibm.com/servers/resourcelink/ They are listed under Hardware products for servers on the Library page. 11.7 References The following publications contain information related to fiber optic link distance: 򐂰 ESCON I/O Interface Physical Layer Document, SA23-0394 򐂰 Coupling Facility Channel I/O Interface Physical Layer, SA23-0395 򐂰 Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367 򐂰 Fiber Transport Services Direct Attach Planning, GA22-7234 For more information about System z connectivity see: http://www.ibm.com/systems/z/connectivity/ For more information about GDPS solutions see: 򐂰 GDPS home page: http://www.ibm.com/systems/z/gdps/ 򐂰 Parallel Sysplex home page: http://www.ibm.com/systems/z/pso/index.html 򐂰 GDPS White paper: http://www.ibm.com/servers/eserver/zseries/library/whitepapers/ 򐂰 GDPS Family - An Introduction to Concepts and Capabilities, SG24-6374 http://www.redbooks.ibm.com/abstracts/sg246374.html?Open Chapter 11. Extended distance solutions 183 5444ch12.fm Draft Document for Review July 21, 2010 6:06 pm For more information about the IBM TotalStorage Proven™ program see: http://www.storage.ibm.com/proven/index.html For information about System z qualified WDM vendor products, see the following Web site: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query="Qualified+WDM"&SearchOrder=1 184 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ax02.fm A Appendix A. CHPID assignment and CHPID mapping tool In previous generations of servers, the CHPID assignments were fixed and could not be changed. System z servers allow you to assign the CHPID number for any feature on the server that would have a CHPID number associated with it. The latest level of the CHPID mapping tool is available to support System z mapping. The CHPID assignment functions are described in this appendix. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 185 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm Description The Channel Path Identifier (CHPID) is used by the IOCDS to define each channel path to a device. The available CHPIDs for a specific server are dependent on the I/O subsystem configuration for that server. Channel CHPID Assignment assigns only the set of CHPID values necessary for each I/O feature. This facilitates utilization of the full set of CHPIDs available to that server. On previous servers, each I/O feature card slot had four CHPIDs assigned. If an I/O feature plugged in the slot used fewer than four CHPIDs, the remainder were not used. These remainder CHPIDs were called “blocked” CHPIDs, and they were used for Internal Coupling (IC) channel definitions. Because of this, in most cases you could not use the full CHPID range (256 CHPIDs) on those servers. A default set of assignments was made when the system were manufactured. If new I/O features were added to the system by upgrades, the same process was used to assign CHPIDs to the new features, leaving the old assignments as they were. If the customer desired, the CHPIDs could be changed when the feature was installed. On System z servers, no default set of CHPID assignments is made at manufacturing time. CHPID mapping is performed using the Hardware configuration Dialog or IOCP, and optionally the CHPID Mapping Tool (CMT). Availability mapping and CHPID assignment for System z servers is described in “CHPID mapping” on page 187. The System z server’s flexible CHPID number assignment facility helps you to keep I/O definitions the same on system upgrades that have different channel card placements or CHPID number assignments. CHPID Mapping Tool (CMT) The CMT is intended for customer use. Since most of the functions and input require a high degree of knowledge regarding the environment, it is extremely difficult for an IBM representative to have the customer’s detailed level of knowledge for the entire configuration. CMT requirements Prior to use of CMT, certain requirements must be satisfied: 1. You must have access to the WWW with a browser that is at least: – Internet Explorer 5.0 – Netscape Navigator 4.7 Your browser must be configured with Java Script and cookies enabled. 2. ResourceLink ID Before using CMT, you must have a valid user ID and password for ResourceLink. The URL for ResourceLink is: http://www.ibm.com/servers/resourcelink There is an option on the Welcome panel to obtain a user ID and password for ResourceLink. Access the Resource Link Web site at www.ibm.com/servers/resourcelink, log in and select Tools  CHPID Mapping Tool. There are several security measures in use by Resource Link that protect your hardware configuration data. 186 IBM System z Connectivity Handbook 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm 3. CCN Number for the new server When the server to be mapped is configured and sent to the manufacturing database from the IBM configurator (e-config), a CCN number is associated with the configuration. This CCN number appears on the output listing of the configurator, and it must be used in order for the mapping function to be able to identify the appropriate server configuration. Users should ask their IBM representatives to verify that they have the latest CCN number associated with the server order. 4. Java Runtime Environment (JRE) The standalone CMT requires the current runtime environment of Java. The Java runtime environment is part of the CMT download. CMT purpose and description The intent of the CMT is to ease installation of new System z servers. It is also intended for making changes to an already installed System z server either to make slight changes to the mapping or as part of an MES action to add or remove channel features on the server. On the System z servers, it requires the use of HCD to accomplish the mapping. There is no task available on the SE to perform this function. CHPID mapping The System z servers do not have default CHPIDs assigned to ports as part of the initial configuration process. It is the customer’s responsibility to perform these assignments by using HCD/IOCP definitions and optionally the CHPID Mapping Tool (CMT). One of the results of using CMT is an IOCP source that will map the defined CHPIDs to the corresponding PCHIDs of the server. There is no requirement to use CMT, you can assign CHPIDs to PCHIDs directly in an IOCP source or through HCD. However, this is a very cumbersome process for larger configurations. If you choose to do manual assignment of CHPIDs to PCHIDs (using HCD or IOCP) it is your responsibility to distribute CHPIDs among the physical channel card ports (PCHIDs) for availability, performance, or both. The objective of CMT is to assist in performing these tasks. Appendix A. CHPID assignment and CHPID mapping tool 187 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm Figure A-1 illustrates the suggested steps used to define a new build for a System z I/O configuration. Assigning PCHIDs CF Report or H/W Config Report for your order (CCN) HCD 1. Create IODF for your server without PCHIDs IODF no PCHIDs 2. Create IODF (validated work IODF) - HCD option 2.12 = Build validated work I/O definition file 3. Create IOCP source without PCHIDs - HCD option 2.3 = Build IOCP input data set 5. Import IOCP source with PCHIDs into IODF - HCD option 5.1 = Migrate IOCP/OS data, Migrate option 3 (PCHIDs) 6. Create a production IODF HCD option 2.1 = Build production I/O definition file CHPID Mapping Tool IOCP source no PCHIDs 4. Run CHPID mapping tool. It will produce an IOCP source with PCHIDs assigned IOCP source with PCHIDs IODF Reports with PCHIDs Figure A-1 PCHID assignment on a new build for a System z server When upgrading it is recommended that you use the CHPID Mapping Tool to configure the CHPID to PCHID assignments. For upgrades, the process shown in Figure A-1 changes in that the IOCP input statements generated for the CMT input must contain the PCHID values previously assigned to CHPIDs in the original server. In this case, CMT assigns new PCHID values to the affected CHPIDs based on the physical movement of the channel features when the appropriate files (CFReport Order file and IOCP file with PCHID assignments from the System z server) have been loaded into CMT. On System z servers, customization information is stored in files on the SE. The files are called configuration files (or config files), and they are named based on the PCHID value (physical location) of the channel. If channels or CHPIDs have associated configuration files, the CHPID Mapping Tool can assign PCHIDs to your logical CHPID definitions or move a CHPID definition to a new location. This can occur regardless of whether channels are moving. The CHPID Mapping Tool can override PCHID assignments for FICON channels supporting FCP and OSA channels. For details regarding this process and the related configuration files, refer to CHPID Mapping Tool User’s Guide, GC28-6825. Mapping function The CMT provides a method for customizing the CHPID assignments for a System z server to avoid attaching critical channel paths to a single point of failure. It should be used after the server order is placed and before the system is delivered for installation. The CMT can also be used to remap CHPIDs after hardware upgrades that increase the number of channel cards and STI links. 188 IBM System z Connectivity Handbook 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm The System z multi-book structure results in multiple MBAs; therefore, there are multiple STI sets. The STI links balancing across a book’s MBAs, I/O cages, and I/O cards is done by IBM at the server’s initial configuration time. Adding a book via MES will result in STI plugging that is different from new build STI plugging with the same number of books. The optional upgrade feature STI Rebalance (FC 2400) can be ordered to replug the STIs as new builds. Because MBA fanout cards used for I/O on a z10 EC and z9 EC are concurrently rebalanced on a model upgrade, FC 2400 for the z10 EC and z9 EC is only used for rebalancing ICBs. Note that when adding a book in configurations where an MBA fanout card has one port connected to the I/O cage and the other used as an ICB, a rebalance will not occur without FC 2400. The concurrent addition of a single book is supported, but be aware that regardless of how the customer planned the previous configuration, the CHPID Mapping Tool (CMT) can be used to evaluate the effects of FC 2400 on the current configuration. 򐂰 If you take the IOCP and the upgrade CFReport without FC 2400 (provided by the IBM Account Representative) and input these via the availability option in the CMT, it will be possible to see any places where a control unit, or group of control units, have single points of failure (SPOFs). 򐂰 For the next step, use the CFReport for FC2400 along with the same IOCP statements and repeat the availability option in the CMT. This will potentially show a different set of SPOFs. By comparing the two reports, you can determine if FC 2400 is the right choice or if any other configuration changes will need to be made. Important: If the z10 EC and z9 EC STI Rebalance feature (FC 2400) is selected at configuration time, and effectively results in STI rebalancing, the server upgrade will be disruptive. The z10 EC and z9 EC STI Rebalance feature will also change the Physical Channel ID number of ICB-4 links, requiring a corresponding update on the server I/O definition via HCD/HCM. FC 2400 can be order to rebalance HCA-1-O fanouts on a System z9. CMT will map the CHPIDs from an IOCP source to Physical Channel Identifiers (PCHIDs) that are assigned to the I/O ports. These PCHID assignments are fixed and cannot be changed. The process used for determining the PCHID assignments is described in the next section. A list of PCHID assignments for each hardware configuration is provided in the PCHID report available when System z hardware is ordered. Unlike previous servers, there are no default CHPID assignments. CHPIDs can be mapped by importing the IOCP source into the CHPID Mapping Tool. The IOCP source must be built for the new hardware order. PCHID assignments The purpose of the CHPID Mapping Tool is to provide a method of assigning CHPIDs to I/O ports in a way that avoids attaching critical paths to a single point of failure. The first step in the process is done by the System z configurator: assigning an identifier to each port that can later be associated with a CHPID. PCHIDs are assigned for you when your hardware is manufactured. 16 PCHIDs are assigned to each card slot. If a 16 port ESCON card were installed in card slot 1 of I/O cage 1, the first port would be assigned PCHID 100, the second port would be assigned PCHID 101, and so on. A two port card would use the first two PCHIDs assigned to its slot and the rest of the 16 PCHID numbers for that slot would be unused. Appendix A. CHPID assignment and CHPID mapping tool 189 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm Both manual mapping and availability mapping require the following input to the CMT: 򐂰 Hardware Configuration report (the output from eConfig; has the extension .CFR). This can be either: – Supplied by your IBM Representative – Downloaded from resourcelink 򐂰 A validated IOCP source For more information about these files and their use, refer to CHPID Mapping Tool, GC28-6825. Manual mapping The manual method can be used to define the relationships between each logical CHPID and physical ports on the server. Availability checking and the accuracy of the mapping with HCD definitions is dependent on the user’s knowledge of the server availability characteristics. However, after you have completed manual mapping and have assigned PCHIDs for all your CHPIDs, you can use the availability function to see if there are any intersects in your mapping. Availability mapping Availability mapping is the recommended method for mapping. This method allows you to input the IOCP source for the proposed server configuration and then define the order in which channel and control unit mapping should be done. This function takes into account the availability characteristics of the server and will ensure that the highest levels of availability will be achieved by assigning channel paths to avoid a single point of failure in that server. While using the tool, you will have the ability to switch between manual and availability mapping. You could, for example, map your CHPIDs with the availability mapping option and then make changes manually. You must provide a copy of the system’s IOCP source and define the priorities for channels and control units in the configuration. The CHPID Mapping Tool then decides how to assign CHPIDs to the I/O ports and provides a CHPID Report for a system that will provide maximum system I/O availability. Maximum availability is achieved by distributing channel paths across different channel cards and STI links. In this way a failure to one channel card or STI link will not affect the availability of a device. Setting control unit priority Priorities are assigned (from 0001 to 9999) for each control unit in the IOCP source. More than one CU can be assigned the same priority. Assigning the same priority to more than one CU means that these units will be mapped together for availability. There are good reasons to do so, including: 򐂰 If one device serves as a backup for another device, you should assign the same priority to both of their control units. 򐂰 All operator or system consoles should have the same priority. 򐂰 If multiple control units are used to provide multiple paths to devices, you should assign the same priority to these control units. When CMT maps control units with the same priority it will assign them with the emphasis on availability. Once you have decided which control units (CUs) should be mapped as a group, you should assign the lowest number priority to the control units in the group that you want to be mapped first. The tool will map the CUs with priority 0001 first. It is advantageous to leave gaps in your 190 IBM System z Connectivity Handbook 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm group numbering (for example, 0010, 0020, 0030, and so on). This allows room for additional groups to be created in the event of a conflict. Any CUs that do not have a priority will be assigned afterwards using the default priority. The default priority is the CUs with the most paths have a higher priority. If multiple CUs have the same number of paths, the CU number is used. That means, a CU with CUNUMBR of 1000 will take priority over a CU with a CUNUMBR of 1100. Once priorities have been entered, the availability mapping can be processed. When the tool completes processing, it will display a list of port assignments that were not included in the IOCP source. This will happen when the new hardware is not the same as the hardware on which the IOCP source was built. Once the priorities have been entered, they can be processed by the CMT. The following options are presented: Reset CHPIDs assigned by Availability Reset CHPIDs assigned by Manual Remap Reset CHPIDs assigned by IOCP Reset CHPIDs assigned for Config Files 򐂰 򐂰 򐂰 򐂰 You can choose to reset any, none, or all of the categories of assigned CHPIDs. Selecting none of the options will only process the unassigned CHPIDs. Attention: Choosing to reset CHPIDs assigned by IOCP may require you to recable your hardware. If you reset CHPIDs assigned for config files, you should have a backup of that data; otherwise, it will have to be re-entered after the upgrade. The priority settings are added to the IOCP source as comments. They will be reused the next time the IOCP source is loaded in the CMT. Intersects The tool displays intersects on the next panel. Intersects are potential availability problems detected by the tool. Reason codes for intersects are provided and explanations of the codes are displayed at the bottom of the panel. Remember that these codes refer to channels in the same group, where this group is defined solely by the same priority codes to the mapping tool. Briefly, the warnings are: D C S M B Assigned Channels are on the same Daughter card. Two or more assigned channels use the same channel card. Greater than half the assigned channels use the same STI. All the assigned channels are supported by the same MBA group. Greater than half the assigned channels are connected to the same book. Note: The CMT will not map the adapter IDs (AID) used for IFB coupling adapters, but will display intersects on AIDs if they exist. These should be considered informational warnings and not errors. Eventually, the tool will run out of highly available places to assign CHPIDs and will resort to plugging more than one control unit on the same channel card, STI, or MBA group. A possible cause for intersects may be that prior mapping to other groups left a small number of unassigned ports. Intersects might be corrected by assigning the group that displays an intersect to a lower numbered priority or dividing the group into smaller groups when possible. If you find an intersect to be Appendix A. CHPID assignment and CHPID mapping tool 191 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm unacceptable and regrouping and reprioritizing does not resolve it, you may need more hardware. CMT output PCHIDs are in fixed physical locations, mapped to CHPIDs by the CMT and assigned to the IOCP. Consequently, the output of the CMT is as follows: 򐂰 The tailored reports. All reports should be saved for reference. The Port Report sorted by location should be supplied to your IBM Service Representative for reference. 򐂰 An IOCP with PCHIDs mapped to CHPIDs by Logical Channel Subsystem. This IOCP can then be migrated back into HCD and a production IODF can be built. Note: If the IOCP source was exported from HCD for CMT input, it must be migrated back into HCD. It cannot be used directly by IOCP. InfiniBand support When planning and configuring InfiniBand connections on a System z server, you must plan for maximum server and device availability. Support for PSIFB links The CMT provides the capability to identify potential availability exposures for Adapter IDs and PORTs. The process to follow is shown in Figure A-2. 1. HCD/HCM definitions to define new PSIFB links in WORK IODF. .CFR file from .CFR file from IBM Resource Link IBM Resource Link 2. Create Validated Work IODF (HCD Option 2.12) and generate stand-alone IOCP (text file). IOCP IOCP (text file) (text file) 3. Invoke CMT availability mapping. Analyze intersects. 4. Check for Intersects. 5. Generate Production IODF. Activate and write IOCDS. Figure A-2 CHPID Mapping Tool process for PSIFB links 192 IBM System z Connectivity Handbook 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm The stages shown in Figure A-2 on page 192 are as follows: 1. HCD or HCM is used to define and connect the CIB CHPIDs to create the PSIFB links. 2. Before a stand-alone IOCP file can be generated, a Validated Work IODF must be generated using HCD option 2.12 as shown in Figure A-3. Activate or Process Configuration Data Select one of the following tasks. 12 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Build production I/O definition file Build IOCDS Build IOCP input data set Create JES3 initialization stream data View active configuration Activate or verify configuration dynamically Activate configuration sysplex-wide Activate switch configuration Save switch configuration Build I/O configuration statements Build and manage S/390 microprocessor IOCDSs and IPL attributes Build validated work I/O definition file Figure A-3 HCD: Building the Validated Work IODF The IOCP file should then be downloaded to the workstation where the CHPID Mapping Tool will be run. The file should be downloaded as TEXT using your 3270 emulation program. 3. Invoke the CHPID Mapping Tool, load the .CFR and import the IOCP file. Perform availability mapping. 4. Analyze the output for intersects across AIDs and PORTs. If no intersects exist, go to step 5. Otherwise, you can change the AID/port assignment in manual mode of the CMT, then rerun availability to check your intersects. When you import the IOCP file back into HCD, it will change the AID/port assigned to the CIB CHPID. a. Disconnect CIB CHPIDs that are to be remapped. This is performed from the CF Channel Path Connectivity List shown in Figure A-4 on page 194. Appendix A. CHPID assignment and CHPID mapping tool 193 5444ax02.fm Draft Document for Review July 21, 2010 6:06 pm CF Channel Path Connectivity List Row 1 of 4 Command ===> ___________________________________________ Scroll ===> PAGE Select one or more channel paths, then press Enter. Source processor ID . . . . . : IB022097 Source channel subsystem ID . : 0 Source partition name . . . . : * / n _ _ _ -------Source-------CHPID Type Mode Occ 01 CIB SHR N 02 CIB SHR N 53 CIB SHR N 54 CIB SHR N -----------Destination-------Proc.CSSID CHPID Type Mode IB012097.0 01 CIB SHR IB012097.0 02 CIB SHR IB042097.0 53 CIB SHR IB042097.0 54 CIB SHR -CUType CFP CFP STP STP Figure A-4 CF Channel Path Connectivity List Specify n next to each CHPID to be disconnected. b. Redefine the CHPIDs across new AIDs and PORTs. c. Reconnect the CIB CHPIDs from the CF Channel Path Connectivity List. At this point, the configuration can be re-validated for intersects by returning to step 2. 5. With all the necessary intersects removed, the production IODF can be built. HCD should be used to perform dynamic reconfiguration to activate the IODF and write the IOCDS. References The following publications contain information related to this appendix: 򐂰 Input/Output Configuration Program User’s Guide for ICP IOCP, SB10-7037 򐂰 Hardware Configuration Definition User’s Guide, SC33-7988 򐂰 IBM zEnterprise 196IBM System z10 Enterprise Class Technical Introduction, SG24-7515 򐂰 IBM zEnterprise 196IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 Server Protocol Planning Guide, SG24-7280 򐂰 Server Protocol Implementation Guide, SG24-7281 򐂰 z/OS Hardware Configuration Definition Planning, GA22-7525 򐂰 z/OS Hardware Configuration User’s Guide, SC33-7988 򐂰 CHPID Mapping Tool, GC28-6825 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 BC Technical Introduction, SG24-7241 The following Web sites contains information about the CHPID mapping tool: http://www.ibm.com/servers/resourcelink http://www.redbooks.ibm.com/abstracts/tips0188.html?Open 194 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ax03.fm B Appendix B. Fiber optic cables This appendix describes the physical attributes of optical fiber technologies supported on System z servers. The following topics are covered: 򐂰 򐂰 򐂰 򐂰 Fiber description Fiber connector types Mode Conditioning Patch (MCP) cables Conversion cables © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 195 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm Description Fiber optic cables utilize light for data transmission, rather than electrical current on copper cables. Fiber optic cables have many advantages; for example, they are many times lighter and have substantially reduced bulk, no pins, a smaller and more reliable connector, reduced loss and distortion, and are free from signal skew or the effects of electro-magnetic interference. Figure B-1 illustrates the two types of optical fiber used in a data center environment, in conjunction with System z servers. These types are: 򐂰 Multimode 򐂰 Single mode The difference between them is the way light travels along the fiber. Multimode has multiple light paths, while single mode has only one light path. Each fiber type consists of three parts: 򐂰 The core can be 50 or 62.5 microns in diameter for multimode, or 9 microns in diameter for single mode. 򐂰 The cladding that surrounds the core is 125 microns in diameter. 򐂰 The outer coating is 250 microns in diameter. Multimode (MM) Fiber Outer coating 250 micron dia. "Multiple paths" for light to travel DATA LED = Light Emitting Diode Cladding 125 micron dia. Core 50 or 62.5 micron dia SX Laser = Short Wavelength Laser Outer coating 250 micron dia. Single Mode (SM) Fiber "Single path" for light to travel DATA LX Laser = Long Wavelength Laser Figure B-1 Fiber optic cable types 196 IBM System z Connectivity Handbook Cladding 125 micron dia. Core 9 micron diameter 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm Connector types for fiber cables System z server channel features use the following fiber cable and connector types: 򐂰 62.5 micron multimode (MM) fiber cable with MT-RJ connector – 16-port ESCON (feature code 2323) – ETR port (z10, z9) 򐂰 9 micron single mode (SM) fiber cable with LC Duplex connector – – – – – – – – – – 2-port Intersystem Channel (ISC-3) (feature code 0217, 0218) 2-port FICON Express4 4 KM LX (feature code 3323) 4-port FICON Express4 10 KM LX (feature code 3321) 4-port FICON Express4 4 KM LX (feature code 3324) 4-port FICON Express2 LX (feature code 3319) 2-port FICON Express LX (feature code 2319) 2-port OSA Express3 10 GbE LR (feature code 3370) 4-port OSA-Express3 GbE LX (feature code 3362) 2-port OSA-Express2 GbE LX (feature code 3364) 2-port OSA-Express GbE LX (feature code 1364) 򐂰 62.5 and 50 micron MM fiber cable with LC Duplex connector – – – – – – – – – 4-port FICON Express4 SX (feature code 3322) 4-port FICON Express2 SX (feature code 3320) 2-port FICON Express4-2C SX (feature code 3318) 2-port FICON Express SX (feature code 2320) 2-port OSA-Express3 10GbE SR (feature code 3371) 4-port OSA-Express3 GbE SX (feature code 3363) 2-port OSA-Express3 GbE SX (feature code 3373) 2-port OSA-Express2 GbE SX (feature code 3365) 2-port OSA-Express GbE SX (feature code 1365) 򐂰 9 micron SM fiber cable with SC Duplex connector – 1-port OSA-Express2 10 GbE LR (feature code 3368) – 2-port OSA-Express GbE LX (feature code 2364) 򐂰 62.5 and 50 micron MM fiber cable with SC Duplex connector 2-port OSA-Express GbE SX (feature code 2365) 򐂰 50 micron MM industry standard OM3 (2000 MHz-km) fiber cable with MPO connector InfiniBand coupling links (feature code 0163) Appendix B. Fiber optic cables 197 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm Figure B-2 shows the most common fiber cable connectors used in data center environments. ESCON Duplex MT-RJ SC Duplex LC Duplex Figure B-2 Most common connectors used for optical cables Mode Conditioning Patch (MCP) cables MCP cables allow for already installed multimode fiber cables to be reused for long wavelength links. The MCP cables are two meters (6.5 feet) long and have a link loss budget of 5.0 dB. The MCPs can be used for the following fiber optic links (Table B-1 on page 199): 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 ISC-3 FICON Express LX FICON Express2 LX FICON Express4 LX OSA-Express GbE LX OSA-Express2 GbE LX OSA-Express3 GbE LX These links have a long wavelength transceiver designed to be used with 9 micron single mode fiber cables. MCP cables allow the long wavelength optical signals to be carried over multimode fiber cables. The ISC-3 feature can only use MCP cabling to operate in compatibility mode (1 Gbps), because 2 Gbps bit rate is not supported with MCP cables. FICON Express4 LX, FICON Express2 LX and FICON Express LX links only support a link speed of 1 Gbps when MPC cables are used. Be aware of the distance reductions when MPC cables are used (Table 11-1 on page 168). Important: One MCP cable must be plugged into the long wavelength transceiver at each end of the link. Note: Fiber optic Mode Conditioning Patch (MCP) cables are not orderable as product feature codes for z10 or z9 servers. 198 IBM System z Connectivity Handbook 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm Fiber optic Mode Conditioning Patch cables can be ordered via the IBM Networking Services fiber cabling services options. See Appendix C, “Fiber cabling services” on page 203. Table B-1 MPC cables MCP Cable description MCP Cable Connector/Receptacle description 9 micron Single Mode to 50 micron Multimode SC Duplex Connector to ESCON Duplex Receptacle 9 micron Single Mode to 50 micron Multimode SC Duplex Connector to SC Duplex Receptacle 9 micron Single Mode to 62.5 micron Multimode SC Duplex Connector to SC Duplex Receptacle 9 micron Single Mode to 62.5 micron Multimode SC Duplex Connector to ESCON Duplex Receptacle ISC-3 Compatibility 9 micron Single Mode to 50 micron Multimode LC Duplex Connector to SC Duplex Receptacle 9 micron Single Mode to 62.5 micron Multimode LC Duplex Connector to SC Duplex Receptacle 9 micron Single Mode to 62.5 micron Multimode LC Duplex Connector to ESCON Duplex Receptacle MCP Cable Connector/Receptacle illustration InfiniBand cables An OM3 50/125 micrometer (2000 MHz-km @ 850 nm) multimode fiber optic cable with Multifiber Push On (MPO) connectors is used for 12x IB-DDR connections. Appendix B. Fiber optic cables 199 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm The cable is an InfiniBand Trade Association (IBTA) industry standard cable. It contains one pair of fibers per lane for the 12x IB-DDR connection (resulting in 24 duplex fibers). The sender and receiver connectors are clearly marked with either RX or TX and the connectors are keyed (Figure B-3). Receiver (RX) Transmitter (TX) Figure B-3 OM3 50/125 micrometer multimode fiber cable with MPO connectors The following standard cable lengths are available from IBM Global Technology Services (GTS): 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 򐂰 10 meters (32.8 feet) 13 meters (42.7 feet) 15 meters (49.2 feet) 20 meters (65.6 feet) 40 meters (131.2 feet) 80 meters (262.4 feet) 120 meters (393.7 feet) 150 meters (492.1 feet) Conversion kits Conversion kits allow for the reuse of already installed cables that are the same fiber optic mode but have different connectors than the ones required (Table B-2). Table B-2 Conversion kit cables Conversion Kit Cable description 200 Conversion Kit Cable Connector/Receptacle description 9 micron Single Mode LC Duplex Connector to SC Duplex Receptacle 62.5 micron Multimode MT-RJ Connector to ESCON Duplex Receptacle IBM System z Connectivity Handbook Conversion Kit Cable Connector/Receptacle illustration 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm Conversion Kit Cable description Conversion Kit Cable Connector/Receptacle description 50 micron Multimode LC Duplex Connector to SC Duplex Receptacle 62.5 micron Multimode LC Duplex Connector to SC Duplex Receptacle 62.5 micron Multimode LC Duplex Connector to ESCON Duplex Receptacle 62.5 micron Multimode LC Duplex Connector to MT-RJ Connector with Coupler 62.5 micron Multimode SC Duplex Connector to LC Duplex Connector with Coupler 9 micron Single Mode SC Duplex Connector to LC Duplex Connector with Coupler Conversion Kit Cable Connector/Receptacle illustration Note: Fiber optic conversion kits are not orderable as product feature codes for z10 or z9 servers. Fiber optic conversion kits can be ordered via the IBM Networking Services fiber cabling services options. Each conversion kit contains one cable. See Appendix C, “Fiber cabling services” on page 203. Appendix B. Fiber optic cables 201 5444ax03.fm Draft Document for Review July 21, 2010 6:06 pm References The following publications contain information related to the topics in this appendix: 򐂰 ESCON I/O Interface Physical Layer Document, SA23-0394 򐂰 Fiber Channel Connection for S/390 I/O Interface Physical Layer, SA24-7172 򐂰 Coupling Facility Channel I/O Interface Physical Layer, SA23-0395 򐂰 Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367 򐂰 S/390 Fiber Optic Link (ESCON, FICON, Coupling Links and OSA) Maintenance Information, SY27-2597 򐂰 Fiber Transport Services Direct Attach Planning, GA22-7234 202 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ax04.fm C Appendix C. Fiber cabling services This appendix describes the IBM fiber cabling services options being offered by IBM Global Technical Services to customers. The following topics are covered: 򐂰 Fiber cabling services options – System z Fiber Cabling Services – Enterprise Fiber Cabling Services 򐂰 Fiber Transport System (FTS) components © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 203 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm Fiber cabling services options When integrating a System z server into a data center, an IBM Installation Planning Representative (IPR) provides planning assistance to customers for equipment power, cooling, and the physical placement of the server. However, the fiber optic cable planning and connecting of the System z server channels to I/O equipment, Coupling Facilities, networks, and other servers, is the customer’s responsibility, both for new server installations and server upgrades. Note: Customers, especially those with complex system integration requirements, may request connectivity assistance from IBM. See the next section for details. Customers with the resources and personnel to plan and implement their own connectivity, or those with less complex system configurations, can consult the following manuals to help them determine and order the required fiber optic cabling. 򐂰 򐂰 򐂰 򐂰 IBM zEnterprise 196 Installation Manual for Physical Planning, GC28-6897 System z10 EC Installation Manual for Physical Planning, GC28-6865 System z9 Installation Manual for Physical Planning, GC28-6844 Planning for Fiber Optic Links, GA23-0367 These manuals are available on the IBM Resource Link Web site: http://www.ibm.com/servers/resourcelink IBM Site and Facilities Services IBM Site and Facilities Services has a comprehensive set of scalable solutions to address IBM cabling requirements, from product-level to enterprise-level for small, medium, and large enterprises. 򐂰 IBM Facilities Cabling Services - Fiber transport system 򐂰 IBM IT Facilities Assessment, Design, and Construction Services - Optimized Airflow Assessment for Cabling Planning and installation services for individual fiber optic cable connections are available. An assessment and planning for IBM Fiber Transport System (FTS) trunking components can also be performed. These services are designed to be right-sized for your products or the end-to-end enterprise, and to take into consideration the requirements for all of the protocols and media types supported on the zEnterprise 196, System z10, System z9, and zSeries (for example, ESCON, FICON, Coupling Links, and OSA-Express), whether the focus is the data center, the Storage Area Network (SAN), the Local Area Network (LAN), or the end-to-end enterprise. IBM Site and Facilities Services are designed to deliver convenient, packaged services to help reduce the complexity of planning, ordering, and installing fiber optic cables. The appropriate fiber cabling is selected based upon the product requirements and the installed fiber plant. 204 IBM System z Connectivity Handbook 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm The services are packaged as follows: 򐂰 Under IBM Facilities Cabling Services there is the option to provide IBM Fiber Transport System (FTS) trunking commodities (fiber optic trunk cables, fiber harnesses, panel-mount boxes) for connecting to the z10 and z9 servers. IBM can reduce the cable clutter and cable bulk under the floor. An analysis of the channel configuration and any existing fiber optic cabling is performed to determine the required FTS trunking commodities. IBM can also help organize the entire enterprise. This option includes enterprise planning, new cables, fiber optic trunking commodities, installation, and documentation. 򐂰 Under IBM IT Facilities Assessment, Design, and Construction Services there is the Optimized Airflow Assessment for Cabling option to provide you with a comprehensive review of your existing data center cabling infrastructure. This service provides an expert analysis of the overall cabling design required to help improve data center airflow for optimized cooling, and to facilitate operational efficiency through simplified change management. FTS connectivity The need for data center cabling implementation arises from the following three scenarios: 򐂰 Establishing a new data center 򐂰 Upgrading an existing data center by replacing the cabling 򐂰 Adding new equipment to an existing data center Appendix C. Fiber cabling services 205 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm There are two choices when implementing fiber optic cabling. The first uses discrete fiber optic jumper cables (Figure C-1). Servers ESCON Directors I/O Figure C-1 Unstructured cable environment Each jumper cable connects one server port directly to another to form a link; for example, one server CHPID port to one ESCON Director port. In today’s data centers, with the huge number of fiber optic cables and their diversity, an underfloor cabling system could soon get out of control and show deficiencies such as: unknown cable routing; no cable documentation system; unpredictable impact of moves, adds, and changes; and presenting unknown risk at every underfloor activity. 206 IBM System z Connectivity Handbook 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm The second choice for ESCON fiber optic cabling is a structured trunking system, illustrated in Figure C-2. Servers Permanent Cabling Patching Area CHPIDS CHPIDS ESCD ESCD ESCD ESCD SWITCH PORTS I/O's CU CU CU CU CU CU CU I/O Zone Central Patching Location Figure C-2 Fiber Transport System in an ESCON environment A structured fiber optic trunking system greatly reduces the number of discrete jumper cables running under the raised floor. FTS offers complete end-to-end connectivity of the fiber optic environment plant for all System z servers link applications. The fiber optic trunk cables connect the server ports to the back of patch panels that are located in the Central Patching Location (CPL). The CPL is usually made up of cabinets or racks that hold the patch panels. The fronts of the patch panels have individual ports that now represent the server ports. Connections between two server ports can be done quickly and easily by running a short jumper cable to connect the two patch panel ports. FTS benefits The most obvious benefit of the structured trunking system is the large reduction in the number of fiber optic cables under the raised floor. The smaller number of cables makes documenting what cables go where much easier. Better documentation means tracing a fiberoptic link is much easier during problem determination and when planning for future growth. A less apparent and often overlooked benefit of a structured system is its ability to make future data center growth implementation much easier. With a structured system installed, channels, ESCON Director ports, and control unit I/O ports are connected by fiber optic trunk cables to patch panels in the CPL. All the connections between the equipment are made with short jumper cables between the patch panels in the CPL. When new equipment arrives, it is connected to patch panels in the CPL as well. Then the actual reconfiguration takes place in the CPL by moving short jumper cables between the patch panels, not by moving long jumper cables under the raised floor. Also, none of the change activity is done near the active equipment, unlike the case with the discrete jumper cable solution. Future equipment additions and changes can be done in the same manner and will not be affected by the amount of equipment already installed on the floor. Appendix C. Fiber cabling services 207 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm Table C-1 shows the advantages of a structured cabling system over a non-structured cabling system. Table C-1 Benefits of the structured cabling system Non-structured cabling Structured cabling Unknown cabling routing Known cable pathways No cable documentation system Defined cable documentation Unpredictable impact of moves, adds and changes Reliable outcome of moves, adds and changes Every underfloor activity is an unknown risk Underfloor activity can be planned to minimize risk FTS is a long-term connectivity solution that provides an organized network of cabling options for future equipment reconfigurations and additions. Changes can be performed with minimal floor disruptions and are accomplished by rearranging short jumpers at the panel-mount boxes in the CPL. Each FTS design is unique in that it is based on physical room characteristics and equipment configurations and placement preferences. FTS provides the following benefits: 򐂰 Proven connectivity: – FTS components are IBM-tested, approved, and sold under the IBM logo. – FTS trunk-mounting kits are designed and tested with System z servers and devices to prevent any impact on equipment operation or serviceability. – Multimode and single mode fiber solutions are available. 򐂰 Elimination of long jumper cables, making underfloor areas less congested and easier to manage by using direct-attach trunking. 򐂰 Cost reductions and productivity gains through: – Faster, less disruptive installation and removal of equipment – Easier reconfiguration of equipment – More efficient use of underfloor space (potential savings in air conditioning requirements) 򐂰 Improved cable management with a reduced risk of damage to fiber cables for the following environments: – ESCON – FICON – Parallel Sysplex – Open Systems Adapter – Open system architectures (Fibre Channel and Gigabit Ethernet) 򐂰 Growth flexibility by being: – Expandable – Relocatable – A long-term solution 208 IBM System z Connectivity Handbook 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm These benefits and potential cost savings must be assessed for each data center and its particular environment. The FTS structured cabling has two implementation solutions. Both use direct attach trunking at the server to connect to the CPL. The main difference is the type of panel-mount boxes used at the CPL. Modular panel-mount connectivity The first FTS implementation solution is called modular panel-mount connectivity. This uses MTP-to-MTP trunk cables to connect to the CPL (Figure C-3). The modular panel-mount boxes are designed to allow you to change the port type in the panel-mount box by unplugging the trunk and changing the modular inserts in the panel-mount box. Benefits: 򐂰 MTP-to-MTP trunks connect quickly to the panel-mount box. 򐂰 The connector type in front of the panel-mount can be easily changed. 򐂰 Panel mount capacity can be customized. ESCON Director Server 72 port Fiber Trunk Cables 72 port ESCON connector panel-mount box 72 port ESCON connector panel-mount box Jumper cable connects CHPID 80 to Port A8 Figure C-3 FTS solution 1: Modular panel-mount connectivity Appendix C. Fiber cabling services 209 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm SC-DC connectivity The second FTS implementation solution is called SC-DC connectivity. It uses MTP-to-SC-DC (Single Connector-Dual Contact) trunk cables to connect to the CPL (Figure C-4). ESCON Director Server 72 port Fiber Trunk Cables 128 port SC-DC connector panel-mount box 128 port SC-DC connector panel-mount box Jumper cable connects CHPID 80 to Port A8 Figure C-4 FTS Solution 2: SC-DC connectivity The SC-DC small form factor optical connector is the standard for FTS connectivity, eliminating the need for multiple connector types at patch panels and for patch cables. By standardizing on a connector with a proven history of reliability, the different optical fiber connectors used in equipment do not create a management problem when a system configuration must change. The SC-DC supports termination of both sizes of multimode fiber, and single mode fiber. As a result, CPL patch panel connections and patch cables are easily managed and differentiated by their color. Benefits: 򐂰 Server ports order at the panel-mount is independent of the harness plugging at the server. 򐂰 Panel-mount ports come factory-labeled with the server port addresses. 򐂰 Single connector type at the CPL. 210 IBM System z Connectivity Handbook 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm Table C-2 shows a comparison between the two types of FTS solutions. Table C-2 FTS connectivity solutions comparison Benefits Jumper cables Modular panel mount connectivity solution SC-DC connectivity solution Organized, manageable underfloor cabling No Yes Yes All server ports located in one central location No Yes Yes System reconfigurations done without lifting floor tiles or opening server covers No Yes Yes Server ports arranged in sequential order and labeled at the panel-mount boxes No No Yes Space allocated for future server growth No No Yes Fiber Quick Connect (FQC) Fiber Quick Connect (FQC) is an option in the eConfig configuration tool when ordering: 򐂰 򐂰 򐂰 򐂰 A new build zEnterprise server A new build System z10 server A new build System z9 server An upgrade of a System z The FQC features are for factory installation of IBM Fiber Transport System (FTS) fiber optic harnesses for connection to all ESCON channels in I/O cages of the z10 and z9 servers and FICON LX on z10. FTS fiber optic harnesses enable connection to FTS direct-attach fiber optic trunk cables. FQC, when coupled with the Fiber Transport System (FTS) products from IBM Global Services, delivers a solution to reduce the amount of time required for on-site installation and setup of cabling, to minimize disruptions, and to isolate the activity from the active system as much as possible. FQC facilitates adds, moves, and changes of ESCON multimode fiber optic cables and FICON LX single mode fibre optic cables in the data center and reduces fiber optic cable installation time. Enterprise fiber cabling services provides the direct-attach trunk harnesses, patch panels, and central patching location (CPL) hardware, as well as the planning and installation required to complete the total structured connectivity solution. CPL planning and layout is done prior to arrival of the server on-site, and documentation is provided showing the channel layout and how the direct attach harnesses are plugged. FQC supports all of the ESCON channels in all of the z10 and z9 servers’ I/O cages. On z10 servers FQC also supports FICON LX channels. FQC cannot be ordered for selected channels and cages within the server. For example, FQC for ESCON is based on a quick connect/disconnect trunking strategy utilizing the 12 fiber MTP connector, so it is able to transport six ESCON CHPIDs. For example, six trunks, each with 72 fiber optic pairs (twelve MTP connectors), can displace up to 420 fiber optic cables, the maximum quantity of ESCON channels supported in one I/O cage on a z10 or z9 server. This significantly reduces ESCON cable bulk. The MTP connector enables Fiber Quick Connect trunk cables to be installed and later disconnected and relocated very quickly. The MTP connector also enables Fiber Quick Connect to bring its fiber trunk cables directly under the covers of the System z servers, and ESCON Directors. Appendix C. Fiber cabling services 211 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm FQC Direct attach harness FICON LX direct attach harnesses have one MTP connector that breaks out to six LC-Duplex connectors. ESCON Direct attach harnesses have one MTP connector that breaks out to six MT-RJ connectors (for 16-port ESCON features), or six ESCON Duplex connectors (for earlier 4-port ESCON features). The direct attach harnesses connect the trunk cable to the individual server ports and accommodate the different industry-standard optical connectors used on System z servers. This is illustrated in Figure C-5. DUPLEX CONNECTOR M TP 12-fiber Connector - ESCON Duplex SC Duplex Multim ode SC Duplex Singlem ode SC-DC Multim ode SC-DC Singlem ode MIC (FDDI) LC-Duplex MT-RJ 1 2 3 4 A lignm ent pins 5 6 Figure C-5 FQC direct-attach harness schematic The number of harnesses required depends on the number of channel ports in the server. MTP coupler brackets The MTP coupler brackets provide the mechanical support to connect the harness MTP connectors to the trunk cable MTP connectors. They are directly mounted on the server frames. Direct attach trunk cable The maximum density of fiber trunk cables is 144 fibers (72 ESCON channels), which use twelve MTP connectors on the server trunk cable end. The trunk cables run under the raised floor and connect the harnesses in the server to the panel-mount box in the Central Patching Location (CPL). Central Patch Location (CPL) panel-mount box The CPL panel-mount box is the connection point for the System z server. This isolates the active server from future system moves, adds, and changes. Note: Although FTS provides cabling solutions for the different industry-standard optical connectors used on System z servers, the Fiber Quick Connect feature, which is the FTS factory-installed direct attach trunking system, is only available for ESCON-attached channels. 212 IBM System z Connectivity Handbook 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm Summary Enterprise fiber cabling services provides a structured FTS fiber cabling system that combines planning, installation, and service. It consists of fiber trunk cables, direct attach harnesses, and a variety of MDF panel mounts. The installation may also include a fiber conveyance system, which is an underfloor tray system that protects the fiber optic cables. Each enterprise fiber cabling services design is unique in that it is based on physical room characteristics and equipment placement preferences. It is a long-term “connectivity solution” that provides an organized network of cabling options for future equipment reconfigurations and additions, such as: 򐂰 Secure configuration environment. 򐂰 Under-floor cable control (long jumper cables can be eliminated, making underfloor areas less congested and easier to manage). 򐂰 Efficient cable management: – Uniform cable documentation – Structured cable routing – Simplified system configuration – Server ports located in one central distribution facility – CHPIDs and device ports in order at patch panels – Factory-installed harnesses and tailgates to ensure consistent routing, allowing quick connection of fiber trunks – Space reserved for future growth 򐂰 Changes can be performed with minimal floor disruptions and are accomplished by rearranging short jumpers at the panel-mount boxes in the MDF. In summary, Enterprise fiber cabling services solutions can provide the following benefits: 򐂰 Complement the ESCON architecture by providing easier copper-to-fiber migration. 򐂰 Cost reductions and productivity gains through: – Faster, less disruptive install and removal of equipment – Easier reconfiguration of equipment – More efficient use of under-floor space (potential savings in air conditioning requirements) 򐂰 Improved cable management with a reduced risk of damage to fiber optic cables for the following environments: – ESCON – FICON – Parallel Sysplex – Open Systems Adapter 򐂰 Growth flexibility by being: – Expandable – Re-locatable – A long-term solution Appendix C. Fiber cabling services 213 5444ax04.fm Draft Document for Review July 21, 2010 6:06 pm These benefits and potential cost savings need to be assessed for each data center and its particular environment. References For further information about the IBM Networking Services fiber cabling services offered by IBM Global Services, and related topics, see: 򐂰 The IBM Resource Link Web site: http://www.ibm.com/servers/resourcelink 򐂰 Fiber Transport Services Direct Attach Planning, GA22-7234 򐂰 Installing the Direct Attach Trunking System in zSeries 900 Servers, GA27-4247 򐂰 ESCON I/O Interface Physical Layer Document, SA23-0394 򐂰 Coupling Facility Channel I/O Interface Physical Layer, SA23-0395 򐂰 Fiber Channel Connection for S/390 I/O Interface Physical Layer, SA24-7172 򐂰 Planning for Fiber Optic Links (ESCON, FICON, Coupling Links, and Open Systems Adapters), GA23-0367 򐂰 Fiber Optic Link (ESCON, FICON, Coupling Links and OSA) Maintenance Information, SY27-2597 214 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ax05.fm D Appendix D. Cryptographic features This appendix briefly describes the external cryptographic features of System z servers, and includes the following topics: 򐂰 򐂰 򐂰 򐂰 Overview Crypto Express2 Crypto Express3 References © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 215 5444ax05.fm Draft Document for Review July 21, 2010 6:06 pm Overview The IBM’s Common Cryptographic Architecture (CCA) defines a set of cryptographic functions, external interfaces, and a set of key cryptographic algorithms (Data Encryption Standard, Advanced Encryption Standard and Public Key Algorithm). These provide a consistent, end-to-end, cryptographic architecture across different IBM platforms and other platforms as well, such as z/OS®, AIX®, IBM i5/OS®, Linux, and Microsoft® Windows® Server 2003. The cryptographic external interfaces in the System z server environment are: 򐂰 CP Assist for Cryptographic Function (CPACF), which utilizes the z196 coprocessor (CoP). Each CoP consists of two compression (and expansion) units, one cryptographic cipher engine, and one cryptographic hash engine. Each cryptographic engine is shared between two processing unit (cores). 򐂰 Crypto Express2: Peripheral Component Interconnect Extended (PCI-X) Cryptographic Coprocessor 򐂰 Crypto Express3: Peripheral Component Interconnect Express (PCIe) Cryptographic Coprocessor The Crypto Express2 and Crypto Express3 combines the functions of PCIXCC (coprocessor) and PCICA (accelerator) in a single feature. The coprocessor function is supported by the Integrated Cryptographic Service Facility (ICSF), a component of z/OS, as well as by the IBM CEX3C Common Cryptographic Architecture Support Program for Linux on System z. The accelerator function is designed for maximum speed Secure Sockets Layer / Transport Layer Security (SSL/TLS) acceleration, rather than for specialized financial applications for secure, long-term storage of keys or secrets. Crypto Express2 Crypto Express2 is supported on System z servers as follows: 򐂰 The Crypto Express2 feature occupies one I/O slot in a z10 EC, z10 BC, z9 EC, or z9 BC server I/O cage or drawer, with a direct connection to a server STI bus through the I/O cage or drawer backplane. 򐂰 The Crypto Express2 feature has two PCI-X adapters1 that combine the functions of either a PCIXCC or a PCICA, with no associated CHPID. Crypto Express2 has two PCHIDs assigned to it according to its physical location in the I/O cage or drawer. 򐂰 There is no need to define a CHPID for the Crypto Express2 feature in the HCD/IOCP. Care must be taken to avoid the use of Crypto Express2 associated PCHIDs by another device in the HCD/IOCP definition. 򐂰 On System z each Crypto Express2 PCI-X adapter can be configured as one of the following: – Secure coprocessor for Federal Information Processing Standard (FIPS) 140-2 Level 4 certification. It includes secure key and clear key functions, and is programmable to deploy standard functions and algorithms using User Defined Extension (UDX). – Accelerator for RSA public key and private key cryptographic operations used with Secure Sockets Layer / Transport Layer Security (SSL/TLS) processing. 1 216 The Crypto Express2-1P feature has one PCI-X adapter. IBM System z Connectivity Handbook 5444ax05.fm Draft Document for Review July 21, 2010 6:06 pm These modes can be configured via the HMC, and the PCI-X adapter must be configured offline to change the mode. Note: When the Crypto Express2 PCI-X adapter is configured as a secure coprocessor, it still provides accelerator functions. However, you can achieve up to three times better performance for those functions if the Crypto Express2 PCI-X adapter is configured as an accelerator. 򐂰 Up to 8 Crypto Express2 features (16 PCI-X adapters per z10 EC, z10 BC, or z9 EC server. 򐂰 Up to 8 Crypto Express2-1P features (8 PCI-X adapters) per z10 BC server. 򐂰 Up to 8 Crypto Express2 features (16 PCI-X adapters) per z9 BC, Model S07 server. 򐂰 Up to 8 Crypto Express2-1P features (8 PCI-X adapters) per z9 BC model S07 server. 򐂰 Up to 4 Crypto Express2 features (8 PCI-X adapters) per z9 BC, Model R07 server. 򐂰 Up to 4 Crypto Express2-1P features (4 PCI-X adapters) per z9 BC, Model R07 server. Note: The Crypto Express2-1P feature is exclusive to the z10 BC, and a minimum of two features must be ordered on all System z servers. Crypto Express3 Crypto Express3 is supported on IBM zEnterprise 196 and on IBM System z10 servers as follows: 򐂰 The Crypto Express3 feature occupies one I/O slot in a z196 or z10 EC I/O cage or in a z196 or z10 BC server drawer, with a direct connection to a server STI bus through the I/O cage or drawer backplane. 򐂰 The Crypto Express3 feature has two PCI Express adapters2 that combine the functions of either a PCIXCC or a PCICA, with no associated CHPID. Crypto Express3 has two PCHIDs assigned to it according to its physical location in the I/O cage or drawer. 򐂰 There is no need to define a CHPID for the Crypto Express3 feature in the HCD/IOCP. Care must be taken to avoid the use of Crypto Express3 associated PCHIDs by another device in the HCD/IOCP definition. 򐂰 On System z each Crypto Express3 PCI Express adapter can be configured as one of the following: – Secure coprocessor for Federal Information Processing Standard (FIPS) 140-2 Level 4 certification. It also includes secure key functions and is programmable to deploy standard functions and algorithms using User Defined Extension (UDX). – Accelerator for RSA public key and private key cryptographic operations used with Secure Sockets Layer (SSL) processing. These modes can be configured via the HMC, and the PCI-E adapter must be configured offline to change the mode. Note: When the Crypto Express3 PCI Express adapter is configured as a secure coprocessor, it still provides accelerator functions. However, you can achieve up to three times better performance for those functions if the Crypto Express3 PCI Express adapter is configured as an accelerator. 2 The Crypto Express3-1P feature has one PCI-X adapter. Appendix D. Cryptographic features 217 5444ax05.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 Up to eight Crypto Express3 features (16 PCI Express adapters per z196, z10 EC, z10 BC server). 򐂰 Up to eight Crypto Express3-1P features (8 PCI Express adapters) per z10 BC server. Note: The Crypto Express3-1P feature is exclusive to the z10 BC, and a minimum of two features must be ordered initially. References For more information refer to the following IBM Redbooks publications: 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 Business Class Technical Introduction, SG24-7241 218 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444ax06.fm E Appendix E. Channel feature attributes This appendix lists the cable types and attributes of channel features supported on System z servers. Not all channel features can be ordered on all server families. Some features are only available when carried forward on a server upgrade. © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 219 5444ax06.fm Draft Document for Review July 21, 2010 6:06 pm Table E-1 lists the cable types and attributes of the channel types supported on System z servers. The maximum number of channels or ports supported based on each server can be found in Table 1-2 on page 10. The footnotes reflect special considerations for some features; however, we recommend checking the referenced chapter for in-depth information. For details on supported extended distances, refer to Chapter 11, “Extended distance solutions” on page 167. Table E-1 System z channel feature support Channel feature Feature codes ESCON 16-port ESCON Connector Cable type Maximum unrepeated distancea Chapter 4, “Enterprise Systems Connection” on page 37 2323b FICON 200 Mbps MT-RJ MM 62.5 µm 3 km (800) Chapter 5, “Fibre Connection” on page 49 FICON Express LX 2319 2 Gbps LC Duplex SM 9 µm 10 km/20 kmc FICON Express SX 2320 2 Gbps LC Duplex MM 62.5 µm MM 50 µm 150 m (200) 300 m (500) 500 m (2000) 1 Gbps LC Duplex MM 62.5 µm MM 50 µm 300 m (200) 500 m (500) 860 m (2000) FICON Express2 LX 3319 2 Gbps LC Duplex SM 9 µm 10 km/20 kmc FICON Express2 SX 3320 2 Gbps LC Duplex MM 62.5 µm MM 50 µm 150 m (200) 300 m (500) 500 m (2000) 1 Gbps LC Duplex MM 62.5 µm MM 50 µm 300 m (200) 500 m (500) 860 m (2000) 4 Gbps LC Duplex MM 62.5 µm MM 50 µm 70 m (200) 150 m (500) 380 m (2000) 2 Gbps LC Duplex MM 62.5 µm MM 50 µm 150 m (200) 300 m (500) 500 m (2000) 1 Gbps LC Duplex MM 62.5 µm MM 50 µm 300 m (200) 500 m (500) 860 m (2000) 4 Gbps LC Duplex SM 9 µm 4 km FICON Express4 SX FICON Express4 4KM LX 220 Maximum bit rate 3322 3324 IBM System z Connectivity Handbook 5444ax06.fm Draft Document for Review July 21, 2010 6:06 pm Channel feature FICON Express4-2C SX Feature codes 3318 Maximum bit rate Connector Cable type Maximum unrepeated distancea 4 Gbps LC Duplex MM 62.5 µm MM 50 µm 70 m (200) 150 m (500) 380 m (2000) 2 Gbps LC Duplex MM 62.5 µm MM 50 µm 150 m (200) 300 m (500) 500 m (2000) 1 Gbps LC Duplex MM 62.5 µm MM 50 µm 300 m (200) 500 m (500) 860 m (2000) FICON Express4-2C 4KM LX 3323 4 Gbps LC Duplex SM 9 µm 4 km FICON Express4 10KM LX 3321 4 Gbps LC Duplex SM 9 µm 10 km/20 kmc FICON Express8 10KM LX 3325 8 Gbps LC Duplex SM 9 µm 10 km FICON Express8 SX 3326 8 Gbps LC Duplex MM 62.5 µm MM 50 µm 21 m (200) 50 m (500) 150 m (2000) 4 Gbps LC Duplex MM 62.5 µm MM 50 µm 70 m (200) 150 m (500) 380 m (2000) 2 Gbps LC Duplex MM 62.5 µm MM 50 µm 150 m (200) 300 m (500) 500 m (2000) OSA Express Chapter 7, “Open Systems Adapter-Express” on page 93 OSA-Express GbE LX 2364 OSA-Express GbE SX 2365 OSA-Express Fast Ethernet 2366 10/100 Mbps OSA-Express 1000BASE-T Ethernet 1366 3362 OSA-Express3 GbE LX OSA-Express3 GbE SX OSA-Express2 GbE LX 3363 3364 1 Gbps SM 9 µm 5 km MCPd 550 m (500) MM 62.5 µm 275 m (200) MM 50 µm 550 m (500) RJ 45 UTP Cat5 100 m 10/100/ 1000 Mbps RJ 45 UTP Cat5 100 m 1 Gbps LC Duplex SM 9 µm 5 km MCP 550 m (500) MM 62.5 µm 275 m (200) MM 50 µm 550 m (500) SM 9 µm 5 km MCPd 550 m (500) 1 Gbps 1 Gbps 1 Gbps SC Duplex SC Duplex LC Duplex LC Duplex Appendix E. Channel feature attributes 221 5444ax06.fm Draft Document for Review July 21, 2010 6:06 pm Channel feature OSA Express2 GbE SX Feature codes 3365 1 Gbps Connector Cable type Maximum unrepeated distancea LC Duplex MM 62.5 µm 275 m (200) MM 50 µm 550 m (500) OSA Express2 1000BASE-T Ethernet 3366 10/100/ 1000 Mbps RJ45 UTP Cat5 100 m OSA-Express3 4P 1000BASE-T Ethernet 3367 10/100/ 1000 Mbps RJ45 UTP Cat5 100 m OSA-Express2 10 GbE LR 3368 10 Gbps SC Duplex SM 9 µm 10 km OSA-Express3-2P 1000BASE-T Ethernet 3369 10/100/ 1000 Mbps RJ45 UTP Cat5 100 m OSA-Express3 10 GbE LR 3370 10 Gbps LC Duplex SM 9 µm 10 km OSA-Express3 10 GbE SR 3371 10 Gbps LC Duplex MM 62.5 µm 33 m MM 50 µm 82 m (500) 300 m (2000) OSA-Express3-2P GbE SX 3373 MM 62.5 µm 275 m (200) MM 50 µm 550 m (500) HiperSockets HiperSockets 10 Gbps LC Duplex Chapter 9, “HiperSockets” on page 137 n/a Coupling links n/a n/a n/a Chapter 10, “Coupling links and common time” on page 149 IC n/a ICB-3 0993 1 GBps FC 0227 10 m ICB-4 3393 2 GBps FC 0228/0229/0230e 10 m ISC-3 (compatibility mode) n/a n/a 1 Gbps 0217f 0218 0219 ISC-3 (peer mode) ISC-3 (RPQ 8P2197 Peer mode at 1 Gbps)g 2 Gbps LC Duplex 1 Gbps n/a SM 9 µm 10 km/12 kmc SM 9 µm MCPh 50 µm 10 km/20 kmc 550 m (500) SM 9 µm 20 kmc PSIFB (HCA2-O) 0163 6 GBps or 3 GBps MPO OM3 MM 50 µm 150 m PSIFB (HCA1-O) 0167 3 GBps MPO OM3 MM 50 µm 150 m PSIFB LR (HCA2-O LR) 0168 5 Gbps or 2.5 Gbps LC Duplex SM 9 µm 10 km 100 kmh Crypto 222 Maximum bit rate Appendix D, “Cryptographic features” on page 215 Crypto Express2 0863 N/A N/A N/A N/A Crypto Express2-1P 0870 N/A N/A N/A N/A IBM System z Connectivity Handbook 5444ax06.fm Draft Document for Review July 21, 2010 6:06 pm Channel feature Feature codes Maximum bit rate Connector Cable type Maximum unrepeated distancea Crypto Express3 0864 n/a n/a n/a n/a Crypto Express3-1P 0871 n/a n/a n/a n/a ETR links ETR link Chapter 10, “Coupling links and common time” on page 149 6154 8 Mbps MT-RJ MM 62.5 µm MM 50 µm 3 km (26 km) 2 km (24 km) a. Minimum fiber bandwidths in MHz/km for multimode fiber optic links are included in parentheses were applicable. b. The ESCON 16-port card feature code is 2323, while individual ESCON ports are ordered in increments of four using feature code 2324. c. Under certain conditions, RPQ 8P2263 may be required in conjunction with RPQ 8P2197. For the z10 BC, RPQ 8P2340 may be required. Check with your IBM representative. d. Mode Conditioning Patch (MCP) cables enable the 1 Gbps single mode cards to connect to multimode fiber. e. FC 0228 cable: z9, z990, z890 to z9, z990, z890 FC 0229 cable: z10 to z9, z990, z890 FC 0230 cable: z10 to z10 f. There are two feature codes for the ISC-3 card: - Feature code 0217 is for the ISC Mother card (ISC-M). - Feature code 0218 is for the ISC Daughter card (ISC-D). Individual ISC-3 port activation must be ordered using feature code 0219. g. RPQ 8P2197 enables the ordering of a different daughter card supporting 20 km unrepeated distance for 1 Gbps peer mode. RPQ 8P2262 is a requirement for that option, and other than the normal mode the channel increment is two, that is, both ports (FC 0219) at the card have to be activated. h. With a DWDM vendor product that is qualified by System z. Appendix E. Channel feature attributes 223 5444ax06.fm 224 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Draft Document for Review July 21, 2010 6:06 pm 5444bibl.fm Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book. IBM Redbooks publications For information about ordering these publications, see “How to get IBM Redbooks publications” on page 226. Note that some of the documents referenced here may be available in softcopy only. 򐂰 IBM zEnterprise System Technical Guide, SG24-7833 򐂰 IBM zEnterprise System Technical Introduction, SG24-7832 򐂰 IBM System z10 Enterprise Class Technical Guide, SG24-7516 򐂰 IBM System z10 Enterprise Class Technical Introduction, SG24-7515 򐂰 IBM System z10 Business Class Technical Overview, SG24-7632 򐂰 IBM System z9 Enterprise Class Technical Guide, SG24-7124 򐂰 IBM System z9 Business Class Technical Introduction, SG24-7241 򐂰 HiperSockets Implementation Guide, SG24-6816 򐂰 OSA-Express Implementation Guide, SG24-5948 򐂰 FICON Planning and Implementation Guide, SG24-6497 򐂰 IBM Communication Controller Migration Guide, SG24-6298 򐂰 IBM Communication Controller for Linux on System z V1.2.1 Implementation Guide, SG24-7223 򐂰 Enterprise Extender Implementation Guide, SG24-7359 Other publications These publications are also relevant as further information sources: 򐂰 Installation Manual-Physical Planning, G5/G6, GC22-7106 򐂰 RMF Report Analysis, SC28-1950 򐂰 Communications Server: IP Configuration, SC31-8513 򐂰 Communications Server: SNA Network Implementation Guide, SC31-8563 򐂰 Communications Server: SNA Resource Definition Reference, SC31-8565 򐂰 Fiber Optic Link Planning, GA23-0367 򐂰 Enterprise Systems Architecture/390 Principles of Operation, SA22-7201 򐂰 FICON I/O Interface Physical Layer, SA24-7172 򐂰 Fiber Transport Services Physical and Configuration Planning, GA22-7234 򐂰 Parallel Sysplex Overview: Introducing Data Sharing and Parallelism in a Sysplex, GC28-1860 © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 225 5444bibl.fm Draft Document for Review July 21, 2010 6:06 pm 򐂰 Setting Up a Sysplex, GC28-1779 򐂰 Parallel Sysplex Systems Management, GC28-1861 򐂰 Parallel Sysplex Hardware and Software Migration, GC28-1862 򐂰 IBM Cabling System Optical Fiber Planning and Installation, GA27-3943 򐂰 Fiber Optic Links (ESCON, FICON, Coupling LInks and OSA) Maintenance Information, SY27-2597 򐂰 Hardware Configuration Definition: User’s Guide, SC28-1848 򐂰 Hardware Configuration Definition Planning, GC28-1750 򐂰 Processor Resource/Systems Manager Planning Guide, SB10-7033 򐂰 lnput/Output Configuration Program User’s Guide for ICP IOCP, SB10-7037 򐂰 z/Architecture Principles of Operation, SA22-7832 򐂰 System z ESCON and FICON Channel-to-channel Reference, SB10-7034 򐂰 OSA-Express Customer’s Guide and Reference for z990, SA22-7935 򐂰 IBM zEnterprise 196 Installation Manual for Physical Planning, GC28-6897 Online resources These Web sites and URLs are also relevant as further information sources: 򐂰 Fibre Channel standards http://www.t10.org http://www.t11.org 򐂰 System z I/O connectivity http://www.ibm.com/systems/z/hardware/connectivity/index.html 򐂰 Parallel Sysplex http://www.ibm.com/servers/eserver/zseries/pso 򐂰 System z networking http://www.ibm.com/systems/z/hardware/networking/ 򐂰 FICON Director vendors http://www.ibm.com/systems/storage/san/enterprise/index.html 򐂰 IBM documentation and tools http://www.ibm.com/servers/resourcelink How to get IBM Redbooks publications You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy books or CD-ROMs, at this Web site: ibm.com/redbooks 226 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444bibl.fm Help from IBM IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services Related publications 227 5444bibl.fm 228 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm 5444IX.fm Draft Document for Review July 21, 2010 6:06 pm Index Numerics 16-port ESCON 44 16-port ESCON channel sparing 46 16-port ESCON feature 44 50.0 micron cable 121, 125 62.5 micron cable 121, 125 A Adapter ID number (AID) 160 adapter interruptions 116 addressing 72 AID 19 ARP offload 108 statistics 108 takeover 108 auto-negotiated 67 availability IP 108 availability mapping 190 B block multiplexer 33–34 broadcast support 107 HiperSockets 144 buffer credits 178 bus and tag 4 bus cable 4 Business Class (BC) 77 byte 34 byte multiplexer 33 C capacity setting A01 12 Cascaded FICON 52 CCN Number 187 Channel CHPID Assignment facility 25 Channel feature 10 Channel Path Identifier 14 channel spanning 21 Channel Subsystem (CSS) 3, 13–14 Channel-to-Channel (CTC) 75 channel-to-channel (CTC) description 84 checksum 109 CHPID 15, 142 number 142 type 2 CHPID Mapping Tool 21, 25 CHPID type FC 75 FCP 75 FCV 78 CHPIDS 75 CMT 21, 25, 186 combined features 12 command mode 63 command mode open exchanges 65 configuration management 24 configuration planning 26 Configurator for e-business 25 Control Unit 14 Control Unit Port (CUP) 72 Coupling Facility 150 configuration alternatives 153 CSS components 14 CTC base sysplex 85 Coupling Facility 85 loosely coupled 85 D data connection isolation 111 data integrity 69 data router 97 DCM 22 definition characteristics 27 Device sharing 59 device types 119 Direct Memory Access 97 Display NPIV Configuration 56 DIX 122–123, 126 DMA 97 dot3StatsTable 106 droop 79, 173, 178 Dynamic Add/Delete logical partition name 16 Dynamic Channel Management 22 dynamic LAN idle 109 Dynamic Virtual IP Addresses (DVIPA) 143 E eConfig 25 EE 114 Enterprise Extender 114 Enterprise Systems CONnection (ESCON) 37 Enterprise Systems CONnectivity (ESCON) 5 equivalent channels 72 ESCON 5, 37–38 Channel-to-Channel 42 converter 42 Director 41 I/O interface 40 link data rate 5 references 47 © Copyright IBM Corp. 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010. All rights reserved. 229 5444IX.fm ESCON channel 40 ESCON control units and devices 42 ESCON conversion to parallel channel 32 ESCON Director 172 ESCON link 40 External Time Reference (ETR) 162 F fabric binding 71 fanout 19 Fanout rebalancing (FC 2400) 162 FC 1366 133 FC 3366 133 FC 9904 56–57 FC-FS 62 FCP concurrent patch 59 description 54 I/O definitions 55 SCSI IPL feature 56 sharing 55 topologies 53 FCP Channel 53 FCP Channel sharing 58 FCP Multipathing 57 FC-SB-4 62 FC-SW 62 FCV 51 Fibre Channel Protocol traffic 75 switch 68 switch fabric and switch control requirements 61 Fibre Channel Physical and Signaling Standard 61 Fibre Channel Protocol (FCP) 6, 52 Fibre Connection (FICON) 6 FICON bridge connectivity 67 cascaded Directors 68 channel 61 description 50 device addressing support 72 distances 177 error recovery function 60 I/O definitions 55 link data rate 6 modes 50 pacing 65 references 81 topologies 52 FICON Express 78 FICON Express2 77 FICON Express4 75 10KM LX feature 76 4 KM LX feature 76 CHPIDs 76 feature 75 FICON Express4 10KM LX 76, 79 FICON Express4 4KM LX 76 FICON Express4 SX 77 FICON Express4-2C 4 KM LX 77 230 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm FICON Express4-2C SX 77 FICON Express8 75 FICON link 66 FICON modes 50 FICON native mode 52 FICON to ESCON conversion function 43, 51, 67, 176 function 132 G GDPS 181 connectivity protocols 183 Get and GetNext request 106 H hardware assists 58, 99, 141 Hardware Configuration Dialog 15, 25 Hardware Configuration Dialog (HCD) 16 Hardware Management Console (HMC) 16 Hardware System Area (HSA) 15, 25 HCA2-O 158 HCA2-O LR 158 HCD 15–16, 25 high integrity fabric 70–71 high performance FICON for System z (zHPF) 62 HiperSockets 2, 8 description 140 functions 140 IPv6 144 Layer 2 support 145 limitations 143 maximum frame size 142 operational flow 141 VLANs 144 HiperSockets Network Concentrator 145 HiperSockets Network Traffic Analyzer 146 HMC (Hardware Management Console) 16 HPDT ATM 100 HPDT MPC 100 HPR 114 HSA 25–26 I I/O cage 133 I/O definition file 25 I/O device 14, 143 data exchange 142 read control 142 write control 142 IBM 9037 Sysplex Timer 150 IBM Personal Communications (PCOMM) 134 IBM Tivoli AF/Remote 134 IBM Tivoli System Automation for z/OS 72 ICB 9 ICB-4 157, 180 Image ID (IID) 16–17 Inbound Workload Queuing (IWQ) 116 InfiniBand coupling links (PSIFB) 152 5444IX.fm Draft Document for Review July 21, 2010 6:06 pm Input/Output Configuration Dataset 25 Input/Output Configuration Dataset (IOCDS) 16 Input/Output Configuration Program 25 insistent domain IDs 71 Integrated Cluster Bus (ICB) 2 Integrated Cluster Bus 4 (ICB-4) 152 Inter System Channels (ISC) 2 interface isolation 112 Internal Coupling (IC) channel 2 Internet Protocol Assist (IPA) 98 Internet Protocol Version 6 144 Intersects 191 InterSystem Channel-3 (ISC-3) 152 intraensemble data network (IEDN) 118 intranode management network (INMN) 118 IOCDS 16, 24–25 IOCP 25 IOCP source 189 IODF 25 IP assist 98 IPv6 support 103 ISC-3 9 ISL 59 ISO/IEC OM3 74 L LAN Channel Station (LCS) 106 large send 104 Layer 2 support 110 LC Duplex 75 LC Duplex connector 75 LIC 138 Licensed Internal Code (LIC) 138 updates 59 link aggregation 111 link incident reporting 60 logical partition 14, 24 logical partition (LP) 75 logical partition (LPAR) 15 logical partition identifier 16 logical unit numbers (LUNs) 54–55 lookup table 140 LP-to-LP communication 98 LUN masking 57 M MAC 108 MAC address 100 Management Information Base (MIB) 106 Maximum IP addresses per OAT 100 Maximum IP stacks and devices 100 Maximum number of MAC addresses 100 Maximum TCP/IP stacks and devices 100 MIF 16 ID 16 MIF (Multiple Image Facility) 43, 72, 119 mode conditioner patch (MCP) 122, 126 mode conditioning patch (MCP) 76 modes of operation 95 Modified Indirect Data Address Word (MIDAW) 66 multicast support 107 Multiple Image Facility (MIF) 16, 43 Multiple Subchannel Sets (MSS) 23 N N_Port ID Virtualization 56, 58–59 name server registration 64 Native FICON 75 non-QDIO 94, 96 NPIV 56, 58 O OAT dynamic 97 open exchange 65 Open Systems Adapter (OSA) 2, 7 Open Systems Adapter for NCP (OSA for NCP) 114 Open Systems Adapter Support Facility (OSA/SF) 95 optimized latency mode (OLM) 110 OSA/SF 95, 101 OSAD 119 OSA-Express 94 cabling requirements 118 devices 119 Network Traffic Analyzer 99 OSA-Express 1000BASE-T settings 123–124, 127 OSA-Express Integrated Console Controller 132 OSA-Express2 94 10 Gigabit Ethernet Long Reach (10 GbE LR) 125 1000BASE-T 123 concurrent LIC update 98 features 121 Gigabit Ethernet short wavelength 126 OSA-Express2 1000BASE-T 132 OSA-Express2 Gigabit Ethernet long wavelength 122 OSA-Express3 132 OSA-ICC 132 base support 132 OSA-ICC software requirements 134 OSC 94 OSD 94 OSE 94 OSM (OSA-Express for management) 94 OSN 94 OSX (OSA-Express for zBX) 94 P Parallel Access to Volumes (PAV) 23 parallel channel 4, 13, 31 control unit types 35 references 35 Parallel Sysplex 9 HiPerLinks 156 Integrated Cluster Bus (ICB) 157 Index 231 5444IX.fm Internal Coupling (IC) Channel 155 References 166 partition number 16–17 PCHID assignment on the z990 server 188 PCHID assignments 189 PCHID report 19, 161 Physical Channel ID (PCHID) 17 Physical Channel Identifiers (PCHIDs) 189 point-to-point 52, 56 PPRC ESCON 173 primary/secondary router function 102 priority queuing 97 program controlled interrupts (PCIs) 110 PSIFB 9, 151, 158 PSIFB coupling link 158 PSIFB LR coupling links 158 purge ARP 108 purge path extended 60 Q QDIO 54, 94, 96 Diagnostic Synchronization 99 QDIO functions 98 QDIO priorities 101 QDIO versus non-QDIO 96 qualification program 181 query and display OSA configuration 118 query ARP 108 Queued Direct Input/Output (QDIO) 54 R reassignment 143 Redbooks Web site Contact us xiv Request Node Identification (RNID) 60 RESOURCE 16 RFC 2355 134 RMF 129 RMF reporting 129 RPQ (Request for Price Quote) 170 S S/390 Open Systems Adapter-Express 94 SA for z/OS 72 SBCON 169 SCSI 53, 58 SCSI IPL 57 SE 16 Server Time Protocol (STP) 151 SIGA 110 Signal Adapter 110 Signal Adapter (SIGA) 140 Simple Network Management Protocol (SNMP) 106 Small Form Factor Pluggable (SFP) 59 SNA mode 100 spanning 21, 72, 119 STI Rebalance 189 232 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm storage area network (SAN) 54 streaming 146 subchannel numbers 23 subchannels 14 support element 16 switched point-to-point 52 Sysplex Timer Time-of-Day 165 system configuration 132 system console 132 System z High Performance FICON 75 T tag cable 4 TCP/IP Passthru 99 thin interrupt 140 TN3270 enhancements (TN3270E) 134 TN3270E emulation 134 TN3270E Server 114 transport mode 63 transport mode open exchanges 65 traps and set 106 type 142 type of traffic 95 U Unrepeated distance 75–76 UPID 16 user logical partition ID 16 V VIPA takeback 102 takeover 102 virtual (VMAC) 100 Virtual IP address (VIPA) 102 virtual IP address (VIPA) 143 virtual LAN 140 Virtual Local Area Network (VLAN) 144 virtual MAC (VMAC) 113 VLAN ID 106 VLAN support 104 GVRP 105 HiperSockets 144 Linux 105 z/OS 105 z/VM 105 W Wavelength Division Multiplexing (WDM) 181 WDM 181 World Wide Names 54 WWPN 56, 59 X XCA 100 Draft Document for Review July 21, 2010 6:06 pm 5444IX.fm Z z BC model R07 12 z10 BC 77 zEnterprise BladeCenter Extension (zBX) 94 zHPF 62 zoning 57 Index 233 5444IX.fm 234 IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm IBM System z Connectivity Handbook IBM System z Connectivity Handbook IBM System z Connectivity Handbook IBM System z Connectivity Handbook IBM System z Connectivity Handbook IBM System z Connectivity Handbook Draft Document for Review July 21, 2010 6:06 pm Back cover ® IBM System z Connectivity Handbook ® The connectivity options available for your data center, and beyond This IBM Redbooks publication discusses the connectivity options available for use within and beyond the data center for the following IBM System z® family of mainframes: zEnterprise 196 (z196) System z10™ Enterprise Class (z10 EC) System z10 Business Class (z10 BC) System z9® Enterprise Class (z9 EC) System z9 Business Class (z9 BC) Comparisons of the supported protocols and architectures 򐂰 򐂰 򐂰 򐂰 򐂰 Guidance for typical usage of the connectivity options This book highlights the hardware and software components, functions, typical uses, coexistence, and relative merits of these connectivity features. This connectivity handbook will assist readers in understanding the connectivity alternatives that are available when planning and designing their data center infrastructure. This book is intended for data center planners, IT professionals, system engineers, technical sales staff, and network planners who are involved in the planning of connectivity solutions for System z servers. INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. For more information: ibm.com/redbooks SG24-5444-11 ISBN 0738433667