Hp Matrix Operating Environment 7.3 Recovery Management User

Transcript

HP Matrix Operating Environment 7.3 Recovery Management User Guide Abstract The HP Matrix Operating Environment 7.3 Recovery Management User Guide contains information on installation, configuration, testing, and troubleshooting HP Matrix Operating Environment recovery management (Matrix recovery management). HP Part Number: 5900-3777 Published: December 2013 Edition: 1 © Copyright 2013 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Acknowledgments Microsoft®, Windows®, and Windows Server® are U.S. registered trademarks of Microsoft Corporation. VMware, VMware Server, GSX Server, ESX Server, and VMotion are trademarks of VMware, Inc. Adobe and Acrobat are trademarks of Adobe Systems Incorporated. Warranty HP will replace defective delivery media for a period of 90 days from the date of purchase. This warranty applies to all Insight Management products. Revision history For supported operating systems, see the HP Insight Management 7.3 Update 1 Support Matrix available at Enterprise Information Library. Document part number Software version Documentation edition Publication date 5900-3777 7.3.0 1 December 2013 Contents 1 Matrix recovery management Overview........................................................5 2 Installing and configuring Matrix recovery management..................................8 Installation and configuration overview........................................................................................8 Installation and configuration prerequisites...................................................................................8 Installing and licensing Matrix recovery management....................................................................9 Uninstalling Matrix recovery management...............................................................................9 Setting up Networking..............................................................................................................9 Setting up Storage..................................................................................................................10 General storage setup notes................................................................................................12 HP P6000 storage setup notes.............................................................................................12 HP P9000 storage setup notes.............................................................................................13 HP 3PAR storage setup notes...............................................................................................13 Creating and installing a User Defined storage adapter..........................................................14 Setting up Local Site logical servers...........................................................................................16 Setting up Remote Site logical servers........................................................................................17 Configuring Matrix recovery management.................................................................................18 Matrix recovery management GUI overview..........................................................................18 Matrix recovery management configuration steps...................................................................19 Matrix recovery management export and import operations....................................................21 DR protection for IO services....................................................................................................23 DR Protection of IO services configuration overview................................................................23 Configure IO properties......................................................................................................24 Configure OO workflow for optional email notification...........................................................26 Network configuration........................................................................................................26 3 Testing and failover operations...................................................................28 Testing Recovery Groups..........................................................................................................28 Failover operations.................................................................................................................29 Planned failover.................................................................................................................29 Unplanned failover............................................................................................................31 Target selection and parallelism during an activation operation....................................................32 4 Dynamic workload movement with CloudSystem Matrix.................................33 Capabilities and limitations......................................................................................................34 Supported platforms...........................................................................................................37 Overview of physical to virtual cross-technology configuration.......................................................37 Configuring logical servers for movement between physical and virtual targets...........................37 Configuring logical servers for movement between dissimilar physical servers............................40 Configuring and managing portable OS images........................................................................40 Portable Images Storage Assistant (PISA)...............................................................................40 Portable Images Network Tool (PINT)....................................................................................41 Configuring and managing cross-technology logical servers.........................................................41 Portability groups...............................................................................................................41 Defining cross-technology logical servers...............................................................................43 Placing a logical server into a portability group................................................................43 Storage definition..........................................................................................................45 Moving between technologies.............................................................................................46 Target attributes.................................................................................................................47 Moving between blade types..............................................................................................48 Managing DR Protected cross-technology logical servers in a Matrix recovery management configuration..........................................................................................................................48 Setting a failover target type preference................................................................................48 Contents 3 5 Issues, limitations, and suggested actions.....................................................50 Limitations.............................................................................................................................50 Hyper-V support limitation for bidirectional configuration.........................................................50 No automatic synchronization of configuration between sites...................................................50 Matrix recovery management job information is not preserved in certain scenarios.....................50 Minor issues..........................................................................................................................50 Firefox browser cannot be used for site export operations........................................................50 ESX configuration setting required for VMFS datastores of Matrix recovery management managed logical servers to be visible at Remote Site............................................................................51 Activation or deactivation job hangs.....................................................................................51 Identical configuration of logical servers between sites............................................................51 One RAID Manager instance per HP P9000 Storage Management Server and One RAID Manager instance per HP P9000 device group...................................................................................52 CLX/HP P9000 software must be installed on a separate Windows system................................52 One active Matrix recovery management configuration operation at any point in time................52 Site delete operation in Matrix recovery management does not remove HP SIM tools..................52 6 Troubleshooting........................................................................................53 Configuration troubleshooting..................................................................................................53 Configuration error messages...................................................................................................55 Warning messages.................................................................................................................58 Matrix recovery management Job troubleshooting.......................................................................59 Failover error messages...........................................................................................................62 Matrix recovery management log files.......................................................................................63 DR Protected IO serivces troubleshooting...................................................................................63 DR Protected IO services configuration troubleshooting...........................................................64 DR Protected IO services failover troubleshooting...................................................................66 Hyper-V based IO service or logical server fails activation............................................................66 7 Support and other resources .....................................................................68 Information to collect before contacting HP.................................................................................68 How to contact HP..................................................................................................................68 Security bulletin and alert policy for non-HP owned software components......................................68 Subscription service................................................................................................................68 Registering for software technical support and update service.......................................................68 How to use your software technical support and update service...............................................69 HP authorized resellers............................................................................................................69 Related information.................................................................................................................69 Matrix recovery management documentation.........................................................................70 Typographic conventions.....................................................................................................70 A Recover the CSV from online pending state..................................................72 B Documentation feeback.............................................................................73 Documentation feeback...........................................................................................................73 Glossary....................................................................................................74 4 Contents 1 Matrix recovery management Overview Matrix recovery management is a component of the HP Matrix Operating Environment that provides disaster recovery protection for logical servers and for Matrix infrastructure orchestration services. Logical servers and Matrix infrastructure orchestration services (IO services) that are included in a Matrix recovery management configuration are referred to as DR Protected logical servers and IO services. A DR Protected logical server can run on a physical machine (c-Class blade) or on a virtual machine hosted by a hypervisor. When a DR Protected logical server is running on a c-Class blade equipped with HP Virtual Connect, it is referred to as a VC-hosted logical server. When a DR Protected logical server is running on a virtual machine under the control of a hypervisor, it is referred to as a VM-hosted logical server. DR Protected IO services can run on VM-hosted logical servers only. A Matrix recovery management configuration consists of two sites, each managed by the HP Matrix Operating Environment. The site where the Central Management Server (CMS) you are logged into is located is referred to as the Local Site, and the other site in the Matrix recovery management configuration is referred to as the Remote Site. Matrix recovery management pairs symmetrically configured logical servers or IO services across the two sites. One logical server or IO service in the pair is activated at one site, while the other (peer logical server or IO service) is deactivated at the other site. The boot images of these DR Protected logical servers and IO services, including applications code and data, reside on disk array volumes. The source volumes at the site where one of the logical servers or IO services in the pair is activated are replicated at the other site where its peer logical server or IO service is deactivated. These volumes are part of a Storage Replication Group that uses storage array supported replication. One or more logical servers or IO services can be associated with a single Storage Replication Group – this is referred to as a Recovery Group. In a Matrix recovery management configuration each Recovery Group has a Preferred Site where you prefer it to be activated, and a Secondary Site where you prefer it to be deactivated. A Recovery Group can only be activated on one site at any time. A set of Recovery Groups that share the same Preferred and Secondary Sites is referred to as a Recovery Group Set—see Figure 1 (page 6). Recovery Group Sets can be selected for activation or deactivation at the Local Site. Recovery Group Sets can be failed over from one site to the other site in a Matrix recovery management configuration. For example, if a disaster occurs at the Local Site, the administrator at the Remote site can trigger a failover for all of the Recovery Groups that were activated at the Local Site by activating them at the Remote site. This prepares the storage associated with the deactivated DR Protected logical servers and IO services at the Remote site for read-write access, and activates those logical servers and IO services. 5 Figure 1 Recovery Group Sets Features and benefits of Matrix 7.3 recovery management • Provides an automated failover mechanism for DR Protected logical servers, DR Protected IO services, and associated storage. • Provides a disaster recovery solution for logical servers and IO services managed by the HP Matrix Operating Environment. NOTE: Supports DR Protection of IO services with virtual server groups only. DR Protection of IO services associated with physical server groups is not supported at this time. 6 • Extends the HP Matrix Operating Environment value of a common management interface to the realm of disaster recovery for logical servers running on physical or virtual machines and for IO services running on virtual machines. • Supports hypervisor hosted virtual machines. • Supports standalone Hyper-V. • Supports Microsoft Windows Server 2008 with Hyper-V R2 and Windows 2012 with Hyper-V in a clustered configuration with CSV. • Supports flexible cross-technology logical servers. VC-hosted logical servers can fail over to become VM-hosted logical servers, and VM-hosted logical servers can fail over to VC hosts. • Supports multiple logical servers and IO services in a Recovery Group. • Supports bidirectional failover of Recovery Group Sets between two sites, allowing both activated and deactivated Recovery Groups to exist at the same site. • Supports HP P6000 Continuous Access Software storage replication in synchronous and asynchronous modes. • Supports HP P9000 Continuous Access Software storage replication in synchronous, asynchronous, and asynchronous journal modes. • Supports HP 3PAR synchronous and asynchronous data replication. • Supports integration with the remote failover features of installed storage replication products, other than HP P6000, HP P9000, or HP 3PAR. These storage replication products are referred to as User Defined storage in Matrix recovery management. Matrix recovery management Overview • Includes Recovery Group startup order settings that let you determine which Recovery Groups are recovered first during a site failover. • Includes a Copy feature that makes it easy to create multiple Storage Replication Groups with the same configuration parameters. • Supports up to 2,500 disaster recovery-protected logical servers in a two-site configuration for VC hosted and VMware VM’s. Of the total of 2,500 logical servers there is a maximum of 500 physical (VC hosted) and 1500 Hyper-V VM hosted logical servers. Examples: 1. 2000 ESX VM hosted logical servers and 500 VC hosted logical servers 2. 2200 ESX VM hosted logical servers and 300 VC hosted logical servers 3. 2500 ESX VM hosted logical serves 4. 1500 Hyper-V VM hosted logical servers 5. 750 Hyper-V IO services each configured with 2 logical servers, 250 VMware IO services each configured with 2 logical servers, and 500 VC hosted logical servers. NOTE: Matrix recovery management supports only virtual machines when provisioning with IO services. • Supports multiple datastores. By reading this HP Matrix Operating Environment 7.3 Recovery Management User Guide, you will gain a better understanding of Matrix recovery management concepts and configuration testing. The Matrix recovery management graphical user interface (GUI), online help, and tooltips provide task-specific guidance. 7 2 Installing and configuring Matrix recovery management This chapter contains sections on Matrix recovery management installation prerequisites, networking setup, storage setup, logical server setup, Matrix recovery management configuration, export and import operations, and DR Protection for IO services. IMPORTANT: If you intend to create DR Protected IO services, see “DR protection for IO services” (page 23) before starting the Matrix recovery management installation and configuration process. Installation and configuration overview The following Matrix recovery management installation and configuration overview includes links to information on each step in the process. 1. Confirm that all Matrix recovery management installation and configuration prerequisites have been met—See “Installation and configuration prerequisites” (page 8). 2. Install and license Matrix recovery management—See “Installing and licensing Matrix recovery management” (page 9). 3. Confirm that you have a supported networking configuration—See “Setting up Networking” (page 9). 4. Configure storage—See “Setting up Storage” (page 10). 5. Configure for DR Protection—See “Setting up Local Site logical servers” (page 16). 6. Configure logical servers at the Remote Site—See “ Setting up Remote Site logical servers” (page 17). 7. Configure Matrix recovery management—See “Configuring Matrix recovery management” (page 18). 8. Configure DR Protection for IO services—See “DR protection for IO services” (page 23) Installation and configuration prerequisites Matrix recovery management is the HP Matrix Operating Environment component that provides the recovery management capability. The HP Matrix Operating Environment and dependent software must be installed on the Central Management Server at the Local Site and the Remote Site before Matrix recovery management can be installed. For more information, see the HP Insight Management 7.3 Installation and Configuration Guide available at Enterprise Information Library. Confirm that both the Local Site and the Remote Site meet the support requirements specified for Matrix recovery management in the HP Insight Management 7.3 Update 1 Support Matrix, including supported servers, storage, browsers, operating systems, databases, and hypervisors. The HP Insight Management 7.3 Update 1 Support Matrix is available at Enterprise Information Library. NOTE: 8 • It is assumed that networking and storage replication links are present between the Local Site and the Remote Site. • When planning for a disaster recovery solution using Matrix recovery management with VMware hypervisor based IO services, you must ensure that the hosts at both the primary and the remote sites match. For example, the hosts with visibility to the replicated volume that the IO service uses and can be selected to activate the replica services at the DR site must be running a version equal to or higher than the primary site host on which the original IO service is created. For example, if primary IO service is created on an ESX 5.5 host, the DR site ESX servers that have the replicated datastore mapped must be running ESX5.5 or later. Installing and configuring Matrix recovery management Installing and licensing Matrix recovery management 1. Install the HP Matrix Operating Environment and dependent software on the Central Management Server (CMS) at the Local Site and the Remote Site. 2. Discover the managed infrastructure at each site from HP Systems Insight Manager. 3. Apply the license for Matrix recovery management using the HP Insight managed system setup wizard. For more information, refer to the HP Matrix Operating Environment 7.3 Getting Started Guide and the HP Insight Managed System Setup Wizard 7.3 Getting Started Guide available at Enterprise Information Library. Uninstalling Matrix recovery management Use the Windows Add/Remove Programs feature as follows: 1. Select recovery management, then click Remove. 2. Wait until the Matrix recovery management product no longer appears in the list. Setting up Networking It is assumed that networking links are present between the Local Site and the Remote Site. You can use Matrix recovery management in a variety of networking configurations, but it is important that you take note of the following Matrix recovery management networking configuration parameters: • In a Matrix recovery management configuration, workloads can be running on DR Protected logical servers at both sites. A Recovery Group can only be activated (workloads running) on one site at any time, but it can be activated at either site such that workloads associated with different Recovery Groups can be running at both sites simultaneously. For this reason, network services such as DNS, DHCP, WINS, and AD must be available locally at both sites. If one site becomes inoperative due to a disaster, network services continue to be available at the other site based on the native disaster recovery capability in these services. This ensures that workloads can be failed over from the failed site to the other site in the Matrix recovery management configuration. Matrix recovery management must not be used to failover network services. NOTE: The Matrix recovery management “startup order” feature is intended to start up critical applications first, not to ensure that startup dependencies are met between applications and infrastructure services such as networking. • Matrix recovery management does not perform DNS updates or update the IP configuration of recovered logical servers during a failover operation. Your Network Administrator is responsible for making the necessary modifications to ensure that network services are available if you configure a logical server to use a different IP or subnet at each site in the Matrix recovery management configuration. • When running on physical targets (VC-hosted) or non VMware ESX virtual targets (VM-hosted), Matrix recovery management does not ensure that logical servers use the same MAC addresses at both sites. When running on VMware ESX hosted virtual targets, Matrix recovery management does ensure that logical servers use the same MAC address at both sites. Your Network Administrator needs to plan for this in the networking configuration for DR Protected logical servers, if you are using DHCP. NOTE: For MAC address details for cross-technology logical servers (logical servers that are capable of running on both VC hosts and VM hosts), see Dynamic workload movement with CloudSystem Matrix. Installing and licensing Matrix recovery management 9 • When running on HP Virtual Connect hosted physical targets, the Portable Images Network Tool (PINT) must be used to prepare the server image to execute on targets with different network interface configurations and MAC addresses. To use PINT, the Local and Remote Sites must be on the same network, and the OS image must be a Linux version that is supported by Matrix recovery management. PINT ensures that the static network configuration from the source server is successfully transferred to the destination server network interfaces, despite the different environment. The executables and README are in the /PI/PINT folder, where is the folder where HP Insight Control server migration is installed. Copy the executable cp011231.exe to the physical server where the image is currently running. Run cp011231.exe to install PINT and start the PINT service. For information on supported Linux versions and HP Insight Control server migration, see the HP Insight Management Support Matrix at Enterprise Information Library. For information on PINT, see the Portable Images Network Tool (PINT) Windows readme or the Portable Images Network Tool (PINT) Linux readme available in the PINT directory on the HP Insight Control DVD at C:\Program Files\HP\Insight Control server migration\PI\PINT. • If the Local Site and corresponding Remote Site managed servers share a common subnet, you must ensure that there is no conflict between MAC addresses assigned by HP Virtual Connect Enterprise Manager (VCEM). For example, if the default address range provided by VCEM is used at both sites, conflict can be avoided by using the VCEM “exclusion ranges” feature. For example, on the Local Site CMS, exclude addresses from 00-21-5A-9B-00-00 to 00-21-5A-9B-FF-FF, and on the Remote Site CMS, exclude addresses from 00-21-5A-9C-00-00 to 00-21-5A-9C-FF-FF. • For DR protected logical servers and IO services, ESX port group names and Hyper-V virtual network names must be identical at the Local Site and the Remote Site. • When configuring ESX networking, the vNetwork Distributed Switch (vDS) are not supported with Matrix recovery management. Make sure to configure the ESX environment with virtual switches only. Setting up Storage Matrix recovery management depends on storage array replication to enable failover of logical servers. It is assumed that storage replication links are present between the Local Site and the Remote Site. HP P6000 Continuous Access Software Disaster Recovery Groups are a good example of this concept. To set up Matrix recovery management Storage Replication Groups: 1. Boot and data LUNs for VC or VM-hosted logical servers that are to be DR Protected must be replicated using supported storage replication methods, for example, HP P6000 Continuous Access, HP P9000 Continuous Access, HP 3PAR, or a User Defined storage replication method. User Defined storage replication is supported when you create and install a storage adapter in your Matrix recovery management configuration for a storage type other than HP P6000, P9000, or HP 3PAR. For example, if you create and install a storage adapter named EMC, the Storage server type drop-down menu for configuring a Storage Management Server includes EMC as a storage server type. For more information see “Creating and installing a User Defined storage adapter” (page 14). 10 • If a DR Protected logical server at the Local Site is VC-hosted, the replicated boot and data LUNs on the array at the Remote Site must be presented to the corresponding recovery logical server. • If a DR Protected logical server at the Local Site is VM-hosted, the replicated boot and data LUNs on the array at the Remote Site must be presented to the VM host (for example, ESX) at the Remote Site that is targeted to run the recovery logical server (for example, ESX guest). Installing and configuring Matrix recovery management NOTE: • Each Recovery Group has a single Storage Replication Group that is used by the logical servers in that Recovery Group only. All boot and data LUNs used by these logical servers must be included in the same Storage Replication Group. • A Storage Replication Group is a set of storage LUNs on a particular disk array that are replicated with write order preserved. This corresponds to the HP P6000 Continuous Access DR Group concept as well as the HP P9000 Continuous Access consistency group concept. • You can configure multiple data stores for a logical server or Matrix infrastructure orchestration service that is planned for disaster recovery with Matrix recovery management. When the Storage Entry types are of FILE or SAN (physical or virtual) type, you must edit the storage pool entry names at the secondary site to match the exact names of the storage pool entries at the original site. For example, in this use case a recovery group is configured with a VM logical server that has a vmfs boot disk and an RDM data disk. Figure 2 Site 1 Storage Screen – original site where VM hosted logical server was first created When creating the logical server at the secondary site, the storage pool entry name will be unique. Whenever there are multiple data stores, you will need to modify the storage pool entry names to match the preferred/primary site storage pool entry names. Figure 3 Site 2 Storage Screen – creating the logical server at the secondary site 2. 3. Create one Storage Replication Group for each set of logical servers that will be included in a Recovery Group. Record the following details about the Storage Replication Group configuration. This information Setting up Storage 11 is required when you configure Matrix recovery management for replicated storage: • Local and Remote Site storage identifiers, for example, an HP P6000 storage array WWN or an HP P9000 array serial number NOTE: In the same way that conflict in the configuration of MAC addresses at the Local and Remote Sites is avoided in “Setting up Networking” (page 9), conflict must also be avoided in the configuration of WWNs, if the WWNs are not private to the respective sites. The same technique using VCEM exclusion ranges is available for array WWN configuration. • Storage Management Server FQDN names and credentials for the Local and Remote Sites, for example, HP P6000 Command View server name and credentials to access the Command View server • Storage Replication Group name given for the boot and data LUNs of the logical servers that will be part of the same Recovery Group, for example, the HP P6000 DR Group name NOTE: • ◦ HP P6000, HP P9000, and User Defined Storage Replication Groups must use the same Storage Replication Group name at the Local Site and the Remote Site. ◦ HP 3PAR remote copy Storage Replication Groups will have different names at the Local Site and the Remote Site. Storage port WWN and LUN for the replicated LUN, in the case of raw LUNs used by DR Protected logical servers General storage setup notes • For information on storage setup of cross-technology logical servers (logical servers capable of being VC-hosted or VM-hosted), see: Dynamic workload movement with CloudSystem Matrix. • For a list of supported storage, see the HP Insight Management 7.3 Update 1 Support Matrix at Enterprise Information Library. • Hyper-V virtual machines in clustered environments must be stored on cluster shared volumes. HP P6000 storage setup notes • With HP P6000 Continuous Access Software storage replication, when an asynchronous replication group is used in a Recovery Group and an unplanned Recovery Group failover occurs, a full copy of the new source vDisks is automatically duplicated on the new destination vDisks when the failed site recovers. If a failure occurs in the middle of this full copy operation, the data on the new destination vDisks could be corrupted. To protect the new destination vDisks, you must enable the HP P6000 Command View auto-suspend setting to prevent an automatic full copy operation from occurring. To protect the new destination vDisks, you must back up the data on them before you manually run a full copy operation. To prevent a full copy of the new source vDisks to the new destination vDisks after the failover of asynchronous Storage Replication Groups, Matrix recovery management automatically sets the mode of all asynchronous Replication Groups to synchronous prior to storage failover and then resets the mode to asynchronous after the storage failover is completed. The storage link must be up and both the Local Site and Remote Site arrays must be managed by the same Command View server for Matrix recovery management to perform this operation. For the failover of asynchronous Storage Replication Groups to succeed, both the Local Site and Remote Site arrays must be managed by the same Command View server. 12 Installing and configuring Matrix recovery management Under rare failure conditions, the mode of a Storage Replication Group can be left in synchronous mode requiring a manual intervention to reset the mode to asynchronous. For more information, see the following: • ◦ HP P6000 Continuous Access Software documentation available at http:// h20000.www2.hp.com. Cick Manuals, then go to Storage→Storage Software→Storage Replication Software→HP P6000 Continuous Access Software. ◦ HP P6000 Command View Software documentation available at http:// h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage Device Management Software→HP P6000 Command View Software. If the password for a Storage Management Server is changed, take the following actions to refresh the Storage Management Server password on the CMS and in the Matrix recovery management configuration: 1. Discover the Storage Management Server with the changed password. 2. Go to the Matrix recovery management user interface Storage Management Servers tab. 3. Select the Storage Management Server that has the changed password, and click on Edit. 4. Select the Refresh SIM Password box and click Save. HP P9000 storage setup notes • • When HP P9000 Continuous Access Software storage replication is used, Matrix recovery management depends on the HP P9000 Cluster Extension Software command-line interface (CLI) to manage storage replication. The HP P9000 Cluster Extension Software CLI must be installed on a standalone Windows system. HP P9000 Cluster Extension Software depends on HP P9000 RAID Manager Software to manage P9000 storage replication. HP P9000 RAID Manager Software instances and configuration files must be configured to manage various device groups that are configured in Matrix recovery management. For more information, see the following: ◦ HP P9000 Cluster Extension Software documentation available at http:// h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage Replication Software→HP Cluster Extension Software. ◦ HP P9000 RAID Manager Software documentation available at http:// h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage Device Management Software→HP P9000 RAID Manager Software. If the password for a Storage Management Server is changed, take the following actions to refresh the Storage Management Server password on the CMS and in the Matrix recovery management configuration: 1. Discover the Storage Management Server with the changed password. 2. Go to the Matrix recovery management user interface Storage Management Servers tab. 3. Select the Storage Management Server that has the changed password, and click Edit. 4. Select the Refresh SIM Password box and click Save. HP 3PAR storage setup notes • When HP 3PAR remote copy replication is used, Matrix recovery management depends on the HP 3PAR Cluster Extension Software command-line interface (CLI) to manage storage replication. The HP 3PAR Cluster Extension CLI in turn depends on the HP 3PAR OS CLI. Both Setting up Storage 13 must be installed on the Central Management Server where Matrix recovery management is installed. For more information, see the following: ◦ HP 3PAR Cluster Extension Software documentation available at http:// h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage Replication Software→HP Cluster Extension Software. ◦ HP 3PAR OS CLI documentation available at http://h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage 3PAR Device Management Software→HP 3PAR Device Management Software. • You need an encrypted password file to manage storage replication on an HP 3PAR storage system. An encrypted password file can be created by running HP 3PAR OS CLI commands. For more information, see the setpassword command in the HP 3PAR OS CLI Reference. The encrypted password file must be present in the STORAGE\3PAR\conf directory where Matrix recovery management is installed. The encrypted password file replaces the need for a user name, domain name, and password required with other types of storage management servers. • The encrypted password file for both the Local Site and Remote Site Inserv storage servers must be available on the CMS at each site, and the name of the password file must be the same on the CMS at each site. • If you upgrade the HP 3PAR OS CLI on the CMS with a software version that is supported by Matrix OE and HP 3PAR Cluster Extension Software, you must modify a property in the Matrix recovery management properties file. Change the OS_CLI_VERSION property in conf\ hp_ir.properties where Matrix recovery management is installed on the CMS. The default value of the property is set to 3.1.2. NOTE: After you install and configure Matrix recovery management at the Local Site, you must manually failover 3PAR remote copy groups to the Remote Site before attempting to configure install and configure Matrix recovery management at the Remote Site. For 3PAR periodic (asynchronous) remote copy, the manual failover action will not synchronize the data in the remote copy group volumes. To prevent data loss, synchronize the remote copy groups before performing stop and failover operations on the remote copy groups. Creating and installing a User Defined storage adapter Matrix recovery management provides a User Defined storage adapter interface specification to enable one-step Matrix recovery management failover capability for storage types that are supported by Matrix OE but not yet integrated with Matrix recovery management. Overview Matrix recovery management can be configured to invoke storage replication management commands for nonintegrated storage when various storage configuration or failover operations are invoked. The User Defined storage adapter specification for nonintegrated storage defines commands to: 14 • Validate Storage Management Server information when Storage Management Servers for nonintegrated storage are configured using the Matrix recovery management GUI. • Validate Storage Replication Group information when Storage Replication Groups that use nonintegrated storage are configured using the Matrix recovery management GUI. • Failover Storage Replication Groups when logical servers that use nonintegrated storage are failed over using the Matrix recovery management Activate operation. Installing and configuring Matrix recovery management Managing nonintegrated storage with Matrix recovery management If your DR Protected logical servers use a nonintegrated storage system that is supported by Matrix OE, and you want Matrix recovery management to automatically invoke storage failover for the nonintegrated storage using the Matrix recovery management Activate operation, you must: 1. Implement and thoroughly test the three commands defined in the “User Defined storage adapter interface specification”, then perform testing at the Local Site and at the Remote Site. 2. Create a new subdirectory under the STORAGE directory where Matrix recovery management is installed. Give the subdirectory a name that identifies the storage type being managed. This name will appear in drop-down menus in the Matrix recovery management GUI. Perform this step at the Local Site and at the Remote Site. NOTE: The name of the subdirectory must be exactly the same at the Local Site and at the Remote Site. 3. 4. 5. 6. 7. 8. Place your implementation of the User Defined storage adapter commands in the subdirectory created for the storage type. Perform this step at the Local Site and at the Remote Site. Define local and remote Storage Management Servers for the nonintegrated storage by selecting the storage type from the drop-down menu in the Storage Management Servers tab in the Matrix recovery management GUI. The storage type in the drop-down menu has the same name as the subdirectory created in step 2. Using management tools for the nonintegrated storage system, create a Storage Replication Group for the storage used by the logical servers that will be DR Protected. Create a Storage Replication Group for the nonintegrated storage by using the Matrix recovery management GUI. The storage type in the drop-down menu has the same name as the subdirectory created in step 2. Create a Recovery Group by using the Matrix recovery management GUI, and associate that Recovery Group with the Storage Replication Group for the nonintegrated storage. Perform a Matrix recovery management Export operation at the Local Site to generate an exportconfig file, then perform an Import operation to import that exportconfig file at the Remote Site. User Defined storage adapter interface specification The following commands are defined in the User Defined storage adapter interface specification: • validatesms.cmd—Validates a Storage Management Server during configuration • validatesrg.cmd—Validates a Storage Replication Group during configuration • failoversrg.cmd—Fails over a Storage Replication Group while Recovery Group activation occurs Command-line arguments • For validatesms.cmd: sms_name= sms_username= • For validatesrg.cmd: sms_name= sms_username= srg_name= local_storage_id= remote_storage_id= Setting up Storage 15 NOTE: Volumes in the Storage Replication Group (identified by srg_name) are replicated between the local storage system (identified by local_storage_id) and the remote storage system (identified by remote_storage_id). • For failoversrg.cmd: local_sms_name= local_sms_username= local_storage_id= remote_sms_name= remote_sms_username= remote_storage_id= srg_name= NOTE: A value of yes requires the Storage Replication Group to be failed over even in cases where the data at the destination site may not be current. A value of no requires storage failover to fail if the data at the destination is not current. Example invocations of Matrix recovery management User Defined adapter implementation • During Storage Management Server configuration in Matrix recovery management: /STORAGE/EMC/validatesms.cmd sms_name=EMC_SMS1 sms_username=admin • During Storage Replication Group configuration in Matrix recovery management: /STORAGE/EMC/validatesrg.cmd sms_name=EMCSE1 sms_username=admin local_storage_id=emc_id1 remote_storage_id=emc_id2 srg_name=SRG1 • During an Activate operation in Matrix recovery management: /STORAGE/EMC/failoversrg.cmd local_sms_name=EMC_SMS1 local_sms_username=admin local_storage_id=emc_id1 remote_sms_name=EMC_SMS2 remote_user_name=admin remote_storage_id=emc_id2 srg_name=SRG1 use_non_current_data=yes Command return code Commands must return 0 on successful completion, and a nonzero error code indicates failure. Multiple User Defined storage adapters Matrix recovery management supports multiple User Defined storage adapters to co-exist in a Matrix recovery management configuration. For each User Defined storage adapter type, you can create a new subdirectory and place your implementation of the three commands for that storage type. For example, to add a storage adapter named EMC, create an EMC subdirectory under /STORAGE and copy all three commands to the newly created directory: • /STORAGE/EMC/validatesms.cmd • /STORAGE/EMC/validatesrg.cmd • /STORAGE/EMC/failoversrg.cmd If your implementation of the storage adapter commands requires passwords to manage storage replication, the storage adapter command implementation must handle passwords securely. It is your responsibility to encrypt/decrypt passwords while saving and retrieving them. Setting up Local Site logical servers The 1. 2. 3. 16 following conditions must be met before you can configure a logical server for DR protection. You must ensure that the logical server is associated with SAN based storage. A logical server must have been activated at least once. An operating system and applications must be installed on a logical server. Installing and configuring Matrix recovery management Logical servers that meet these conditions appear in the Available LS(s) list in the Local Site Matrix recovery management configuration GUI, even if they are deactivated after they have been activated for the first time. For sites with a large number of logical servers, partitioning logical servers into portability groups can reduce activation time during failover. HP recommends that the portability group associated with logical servers on both the Local Site and the Remote Site be limited to a subset of virtual machine hosts and virtual connect blades that are capable of hosting these logical servers. For additional information on configuring portability groups, see Logical servers→Menus & screens→Manage portability groups in the HP Matrix OE Visualization and Logical Servers online help. NOTE: You cannot change the datastore of a VM-hosted logical server while it is being managed by Matrix recovery management. To change the datastore, first remove the VM-hosted logical server from the Matrix recovery management configuration, then use the Logical Servers Activate operation in the Tools menu of the Visualization tab to change the datastore. After the datastore has been changed, follow the steps in “Setting up Storage” (page 10), “Setting up Networking” (page 9), and “ Setting up Remote Site logical servers” (page 17) to re-add the logical server to the Matrix recovery management configuration. Setting up Remote Site logical servers 1. 2. 3. Deactivate and gracefully shut down the logical servers in the Recovery Groups at the Local Site using Matrix OE visualization. Using the appropriate storage management tools (for example, HP P6000 Continuous Access Software), fail over the storage associated with the Recovery Groups to the Remote Site. If there are VM-hosted logical servers that will be DR Protected: a. Rescan storage using VM host management tools, for example, VMware Virtual Center or Microsoft Hyper-V Management Console, to ensure that the VM host recognizes the failed over storage. • The replicated disk on the Remote Site Hyper-V host must be configured with the same drive letter that is assigned to the Local Site disk it is replicated from. In the case of a cluster shared volume, the replicated disk on the Remote Site Hyper-V host must be configured with the same volume path that is assigned to the Local Site disk it is replicated from. In the case of a cluster shared volume or shared cluster disks, the replicated disk on the Remote Site Hyper-V host must be configured with the same cluster resource name that is assigned to the Local Site disk it is replicated from. b. 4. • Storage pool entry names must be the same at both the preferred and secondary site. • You can configure multiple data stores for a logical server or Matrix infrastructure orchestration service that is planned for disaster recovery with Matrix recovery management. • The names of recovery logical server storage entries must be the same as the names of the logical server storage entries on the Local Site. • The names of recovery logical server storage pool entries must be the same as the names of the logical server storage pool entries on the Local Site. Refresh Virtual Machine resources using the Logical Servers Refresh operation in the Tools menu of the Visualization tab in Matrix OE visualization. Use Matrix OE visualization to create recovery logical servers. Specify the replicated LUNs information as part of logical server creation. When entering storage information during Setting up Remote Site logical servers 17 VM-hosted logical server creation, select the datastore name of the datastore where the VM-hosted logical servers were created. NOTE: • Do not activate the recovery logical servers at this time. During the Matrix recovery management configuration process, the recovery logical servers will be further configured in the configuration import process at the Remote Site and there will be an opportunity to activate and deactivate the recovery logical servers at that time. • To avoid confusion, HP recommends adopting a best practice of using the same logical server name at the Remote Site as the one that was used for the associated logical server at the Local Site. • For information on cross-technology logical servers (logical servers capable of being VC-hosted or VM-hosted), see Dynamic workload movement with CloudSystem Matrix. • For sites with a large number of logical servers, partitioning logical servers into portability groups can reduce activation time during failover. HP recommends that the portability group associated with logical servers on both the Local Site and the Remote Site be limited to a subset of virtual machine hosts and virtual connect blades that are capable of hosting these logical servers. For additional information on configuring portability groups, see Logical servers→Menus & screens→Manage portability groups in the HP Matrix OE Visualization and Logical Servers online help. Configuring Matrix recovery management After you install Matrix recovery management, you can launch the Matrix recovery management GUI from the HP Matrix Operating Environment home page by selecting Tools and then selecting Matrix recovery management... from the drop-down menu. Use the Matrix recovery management GUI to configure Matrix recovery management, manage DR Protected logical servers and DR Protected IO services, and test failover capability. Matrix recovery management GUI overview The home screen for the Matrix recovery management user interface includes tabs for configuration and administration tasks—see Figure 4 (page 18). Figure 4 Matrix recovery management home screen 18 Installing and configuring Matrix recovery management Matrix recovery management user interface tabs • Home Information on the most recent Matrix recovery management Job is displayed at the top of the Matrix recovery management Home screen, including the Latest Job status:, the Job Id:, the Start Time: (if the job is in progress), or the End Time: (if the job has completed). This is followed by a list of the other Matrix recovery management configuration tabs with Configured or Not configured icons to indicate if the configuration tasks for each tab have been completed. • Sites Configure Preferred and Secondary Sites. Activate or Deactivate Recovery Groups. Edit or delete existing Site configurations. Export or import Site configurations. • Storage Management Servers Define new Storage Management Servers. View, edit, or delete Storage Management Servers. • Storage Replication Groups Create new Storage Replication Groups. View configuration details for Storage Replication Groups. Copy, edit, or delete Storage Replication Groups. • Recovery Groups Create new Recovery Groups. View configuration details for Recovery Groups. Import, edit or delete Recovery Groups. Enable or Disable Maintenance Mode on Recovery Groups. • Jobs Monitor Job progress and view Job details. Cancel Jobs in progress. Restart failed Jobs. View logs for Jobs and Sub Jobs. Delete Job information. The Matrix recovery management online help and tooltips provide answers to questions you may have while using the GUI. Matrix recovery management configuration steps Figure 5 (page 20) illustrates the six-step Matrix recovery management configuration process. After the Matrix recovery management configuration process is completed at the Local Site, Matrix recovery management must be configured at the Remote Site. To simplify the Remote Site configuration process and to help ensure that the two sites have synchronized configurations, you can export the Matrix recovery management configuration information to a file at the Local Site, move that file to the Remote Site, and then import the Matrix recovery management configuration information at the Remote Site. Configuring Matrix recovery management 19 Figure 5 Configuration steps NOTE: Make sure all of the steps in “ Setting up Remote Site logical servers” (page 17) have been completed, before you attempt the following Matrix recovery management configuration steps at the Local Site. 1. 2. 3. 4. From the Sites tab, configure the Local Site. From the Storage Management Servers tab, configure Storage Management Servers at the Local Site. From the Storage Replication Groups tab, configure Storage Replication Groups at the Local Site. From the Recovery Groups tab, configure Recovery Groups at the Local Site. NOTE: Matrix recovery management allows failover of the logical servers or the IO services in a Recovery Group, independent of the associated Storage Replication Groups. This capability is referred to as Storage Decoupling. 5. 6. 7. 8. 9. 20 From the Sites tab, create an export file at the Local Site. For information on export and import parameters, see “Matrix recovery management export and import operations” (page 21). From the Sites tab at the Remote Site, import the Local Site Matrix recovery management configuration. For more information, see “Matrix recovery management export and import operations” (page 21). Test the recovery logical servers. All imported Recovery Groups are in maintenance mode, allowing activation of the recovery logical servers. For more information, see “Testing Recovery Groups” (page 28). Deactivate the recovery logical servers and disable Maintenance mode at the Remote Site — for more information see “Testing Recovery Groups” (page 28) Fail back the Storage Replication Groups to the Local Site, then activate the Local Site logical servers. If there are VM-hosted logical servers, use the VMware Virtual Center or the Microsoft Hyper-V Management Console to rescan and refresh virtual machine resources. Installing and configuring Matrix recovery management Matrix recovery management export and import operations This section lists key points about Matrix recovery management export and import behavior. Export • The Matrix recovery management configuration at the exporting site is included in the exportconfig file generated at the exporting site. The Matrix recovery management configuration export file is named exportconfig by default. The exportconfig file is saved to a default location specified by the browser on the Administrator's system. You have the option to change the default export configuration file name, for example, if you want to save multiple Matrix recovery management configuration export files. You can also change the location that the export configuration file is saved to, before completing the save operation. • All of the Recovery Groups that you want to export from the Local Site and import to the Remote Site must be activated at the Local Site. Recovery Groups that are deactivated when the export operation is performed at the Local Site are not imported at the Remote Site. If there are no activated Recovery Groups at the exporting site, the generated exportconfig file cannot be used at the importing site. • DR Protected IO services belonging to Recovery Groups are exported, and they can be imported at a Remote Site, however, a replica IO service cannot be exported or imported. Import • Only Recovery Groups that are activated at the exporting site are imported. • The Matrix recovery management import operation automatically creates a replica IO service for each IO service belonging to each IO services Recovery Group being imported. NOTE: A replica IO service from a Remote Site cannot be imported using a Matrix recovery management import operation. • If the importing site is not configured for Matrix recovery management, Matrix recovery management Site information is not imported. • If the importing site is configured for Matrix recovery management, the site name and CMS at the importing site must match the site name and CMS in the exportconfig file. • Storage Management Server information is imported if Storage Management Servers are not configured at the importing site. If the importing site already contains Storage Management Server information: ◦ The site locality of each Storage Management Server configured at the importing site must match the site locality of a Storage Management Server in the exportconfig file. ◦ The type of Storage Management Server must match if the name matches. ◦ For HP P6000, the user name and port number fields must match. ◦ For HP P9000, the user name and RAID instance number must match. ◦ For HP 3PAR storage system, the password file name must match. If any one of the above items is not matched between the exporting site and the importing site, the import operation fails. NOTE: To manage HP 3PAR remote copy, the encrypted password file for both the Local Site and Remote Site Inserv storage servers must be available on the CMS at each site, and the name of the password file must be the same on the CMS at each site. • Storage Replication Group information associated with activated Recovery Groups in the exportconfig file is imported if the importing site has no Storage Replication Group configuration. Configuring Matrix recovery management 21 • If the importing site already contains Storage Replication Group information: ◦ If the Storage Replication Group is included in another Recovery Group that belongs to the same Recovery Group Set, the import is allowed. ◦ If the Storage Replication Group is included in another Recovery Group that belongs to a different Recovery Group Set, the import fails. It does not matter whether this Recovery Group is activated or deactivated at the importing site. • All entities referenced by imported Recovery Groups are imported, including Sites, Storage Management Servers, and Storage Replication Groups. At the end of an import, only the imported Recovery Groups and their associated entities remain. Storage not attached to the activated Recovery Groups being imported is deleted. • If a Recovery Group in the exportconfig file has the same name as a Recovery Group at the importing site, but it belongs to a different Recovery Group Set, the import fails. • Storage Management Servers and Storage Replication Groups that are not referenced by the imported Recovery Groups are not imported. • If the Recovery Groups in the exportconfig file already exist at the importing site (Recovery Groups with the same name are at the importing site), they are deleted and replaced with the imported Recovery Groups. The Storage Replication Groups referenced by Recovery Groups that are deleted are also deleted. Single Recovery Group import If you add or modify (edit) a Recovery Group at a Matrix recovery management site, there is a single Recovery Group import feature that saves time by propagating those changes to the other site without performing a site import operation. Ensure that the Storage Replication Group associated with the new Recovery Group already exists, or you must manually create it at the DR site before performing the import operation. For example, if you add or modify (edit) a Recovery Group at the Local Site, you can import the new or modified Recovery Group to establish its peer Recovery Group at the Remote Site without importing the entire Local Site configuration. The procedure at the Local Site is the same as the site export procedure. From the Sites tab at the Local Site you export the Matrix recovery management configuration to an exportconfig file and move that file to the Remote Site. At the remote site, browse to the Storage Replication Groups tab and ensure that the Storage Replication Group already exists, or create it manually. From the Recovery Groups tab at the Remote Site, click import...→Select import file.... The Import Wizard window appears allowing you to select a Recovery Group to import from the exportconfig file. See the Matrix recovery management online help for more information on the import procedure for single Recovery Groups. NOTE: 22 • Recovery Groups can be imported one at a time only. You must repeat the import...→Select import file... procedure for each Recovery Group that you import. • If a Recovery Group in the exportconfig file has the same name as a Recovery Group at the importing site, it is not imported. If you are importing a Recovery Group that already exists in the Matrix recovery management configuration but has been modified, you will need to delete the Recovery Group of the same name at the importing site, before you can import the modified version of that Recovery Group. • If a Recovery Group in the exportconfig file references a Storage Replication Group that is being used by another Recovery Group at the importing site, the import fails. Installing and configuring Matrix recovery management DR protection for IO services Matrix infrastructure orchestration (IO) provides rapid provisioning and re-purposing of infrastructure services from shared resource pools using a self-service portal. IO delivers advanced template-driven design, provisioning, and ongoing operations for multi-node, multi-tier infrastructure services. Beginning with Matrix Operating Environment 7.1, IO is integrated with Matrix recovery management to provide DR Protection for IO services. IO services are capable of DR Protection in a Matrix recovery management configuration when they are: • Deployed on virtual machines from an IO template • Associated with storage that is supported by Matrix recovery management (this includes User Defined storage—for more information see “Creating and installing a User Defined storage adapter” (page 14)) IO services that are capable of being DR Protected in a Matrix recovery management configuration are referred to as recoverable IO services. Recoverable IO services can be selected from a drop-down menu in Matrix recovery management when you configure IO services Recovery Groups. After IO services Recovery Groups are configured, the IO services they contain will be DR Protected. OO workflows provide system administrators with automatic email notification when IO operations (for example, create service, delete service, add disk, add server, change lease) are performed on DR Protected IO services that require configuration changes in Matrix recovery management. When Matrix recovery management and IO are installed on the same CMS and recoverable IO services have been configured, you can create DR Protected IO services Recovery Groups as part of the initial Matrix recovery management configuration process, or you can add DR Protected IO services Recovery Groups to an existing Matrix recovery management configuration. The configuration procedure is similar. DR Protection of IO services configuration overview 1. 2. 3. 4. 5. Configure IO properties on Local Site and Remote Site CMS. For more information, see “Configure IO properties” (page 24). Configure the OO workflows. For more information, see “Configure OO workflow for optional email notification” (page 26). Configure network for DR Protected IO services. For more information, see “Network configuration” (page 26). Create recoverable IO services templates. For more information, see the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. Configure recoverable IO services using IO. For more information, see the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. DR protection for IO services 23 NOTE: 6. 7. 8. 9. • When you use Matrix recovery management to DR protect VMware ESX based IO services that have been deployed from an IO template using an ICVirt template or a VM template on the vCenter server at the Local Site, an ICVirt or VM template with same name must be available at the Remote Site before you perform a Matrix recovery management import operation to import the site configuration. If an ICVirt or VM template with the same name is not available at the Remote Site, DR protected IO services will not be imported. • When you use Matrix recovery management to DR protect Microsoft Hyper-V based IO services that have been deployed from an IO template using an ICVirt template on the CMS at the Local Site, an ICVirt template with same name must be available at the Remote Site before you perform a Matrix recovery management import operation to import the site configuration. If an ICVirt template with the same name is not available at the Remote Site, DR protected IO services will not be imported. • When you use Matrix recovery management to DR protect IO services deployed from an IO template using an ICVirt template, VM template, or SCVMM template, the template may contain only a single disk. Add IO services to Matrix recovery management Recovery Groups. Recoverable IO services can be selected from a drop-down menu when you configure IO services Recovery Groups. For more information, see the Recovery Groups tab in the Matrix recovery management GUI. Duplicate the organizational setup of the local site on the remote site. Export site configuration at Local Site. For more information, see “Matrix recovery management export and import operations” (page 21). Import site configuration at Remote Site. For more information, see “Matrix recovery management export and import operations” (page 21). Configure IO properties To enable DR Protection for IO services, edit the following hpio.properties and dr.properties files located in C:\Program Files\HP\Matrix infrastructure orchestration\conf directory: hpio.properties • Enable DR Protection for IO services: enable.dr.protection = true • For DR protection in a multi-tenancy environment, all IO services must have unique names across all organizations. Enforce this restriction by setting the force.unique.service.names.across.organizations=true property. • Specify the datastore used for provisioning DR Protected IO services: ◦ At the Local Site, specify the volumes to use for DR Protected IO services, for example: volume.dr.list = /vmfs/volumes/ds_1;C:\\ClusterStorage\\Volume3;C:\\ClusterStorage\\Volume4 ◦ At the Remote Site, specify the volumes to use for replica IO services, for example: volume.dr.replica.list = /vmfs/volumes/ds_1;C:\\ClusterStorage\\Volume3;C:\\ClusterStorage\\Volume4 24 Installing and configuring Matrix recovery management NOTE: If one datastore is specified in volume.dr.list, the DR Protected IO services are provisioned on the datastore specified. If multiple datastores are specified in volume.dr.list, the DR Protected IO services are provisioned on the datastore in volume.dr.list that is both available for that server pool and also has the most free disk space. If multiple datastores are specified in volume.dr.list, and the IO template specifies one of the datastores in volume.dr.list, the DR Protected IO services are provisioned on the datastore specified in the IO template. • Set the static.ip.replica.list property to reserve static IP addresses for replica IO services at the Remote Site. The replica IP range can be a subset of the IP range configured on IO networks UI. The IP range specified in the property file, reserved for replica list cannot be used when local services are created at the DR site. For example, if the networks configured in IO at the Remote Site are subnetX and subnetY, one or more IP addresses can be reserved for replica services from those subnets. For example, reserve a new IP range for DR protected servers by setting the property in hpio.properties file, similar to the following: static.ip.replica.list = subnetX:192.16.0.10;subnetX:192.16.0.15-192.16.0.20;subnetY:192.16.0.30-192.16.0.40;subnetX:1024:3143::c425:b0e1:e1c0-1024::3143:c425:b0e1:e1cf dr.properties • On the Local Site and Remote Site CMS specify a name identifying the associated site where IO is running in the dr.properties file, for example: local.site = siteA at the Local Site, and local.site = siteB at the Remote Site. The name in the dr.properties file can be different than the site name configured in Matrix recovery management. • If the service owner domain name and username on the Local Site and Remote Site are not the same, set the owner.username.. = owner.username.. property to specify the service owner domain and username mapping. For example, if the service owner domain name and username configured for IO on the Local and Remote Sites are domainA\userA and domainB\userB: owner.username.siteA.domainA\\userA = owner.username.siteB.domainB\\userB Specify additional owner-user name mapping by adding to the dr.properties file. For example: owner.username.siteA.ARCMS9671v\\Administrator = owner.username.siteB.ARCMS9771V\\Administrator owner.username.siteA.ARCMS9671v\\svcuser = owner.username.siteB.ARCMS9771V\\svcuser • If the domain names on the Local Site and Remote Site are not the same, set the owner.domain.. = owner.domain.. property to specify the domain name mapping. For example, if the domain name configured for IO on the Local and Remote Sites is domainA and domainB: owner.domain.siteA.domainA = owner.domain.siteB.domainB • Map the organization administrator in the dr.properties file by setting the owner.username..\\ = owner.username..\\ property. For example, if the organization administrator for IO on the Local and Remote Sites is userA: owner.username.siteA.domainA\\userA = owner.username.siteB.domainB\\userA • If the HPIO property federated.io is set to true in the hpio.props file, the dr.props file must have a corresponding mapping entry: DR protection for IO services 25 federated.siteA. = federated.siteB. For example, if the CMS name at the local site was hostA.test.net and at the remote site it was hostB.test.net, the mapping line would be: federated.siteA.hostA.test.net = federated.siteB.hostB.test.net NOTE: Use this mapping whenever federated.io is set to true, even if federation is not used. For more information, see the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. Configure OO workflow for optional email notification You have the option to configure OO workflows to provide system administrators with automatic email notification when IO operations are performed on DR Protected IO services that require configuration changes in Matrix recovery management (for example, create service, delete service, add disk, add server, change lease). IR delivers the IRWorkflow.zip and Send DR Config Email.xsl files to enable automatic email notification. These two files can be found in the C:\Program Files\HP\Insight Recovery\conf\OO\repo folder on the CMS. The IRWorkflow.zip file is the OO repository export zip file containing the IR workflow and the new OO system property named HpioDrServiceActionRecipients. The Send DR Config Email.xsl file is used by the IR workflow to build the body of the DR email notification. 1. On the Local and Remote Site CMS, import the workflow packaged in the IRWorkflow.zip file using OO Studio. The workflow is imported as DR Global Service End Action under the Library/Hewlett-Packard/Infrastructure orchestration/Service Actions/DR folder in the OO repository. 2. Copy the Send DR Config Email.xsl file to the C:\Program Files\HP\Matrix infrastructure orchestration\conf\OO directory on the CMS. 3. Configure OO to set up email notification. For information on setting up email notification see the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. 4. Edit the HpioDrServiceActionRecipients OO system property to specify email recipients for notification of changes to DR Protected services. 5. Modify the following property in the C:\Program Files\HP\Matrix infrastructure orchestration\conf\hpio.properties file to point to the IR workflow: oo.global.service.end.action.path = Service Actions/DR/DR Global Service End Action 6. Restart Matrix infrastructure orchestration. Network configuration To allow the same IP addresses for primary and replica IO services, when the subnet is spanned between the Local and Remote Site: • The Local Site and the Remote Site must define the same static IP range and it must contain the IP range of the primary IO service. At the site with the replica IO service, the IO administrator must specify a list or a range of IPs (IP exclusion list) in the hpio.properties file to avoid IP address conflicts. These IP addresses will be reserved for replica IO services only and cannot be used by any other IO services. If the subnet is not spanned between the Local and Remote site, the primary and replica IO services can be assigned IP addresses specific to each site. 26 Installing and configuring Matrix recovery management NOTE: Matrix recovery management does not perform DNS updates or update the IP configuration of logical servers associated with IO services during a failover operation. Your Network Administrator is responsible for making the necessary modifications to ensure that network services are available if you configure a logical server to use a different IP address or subnet at each site in the Matrix recovery management configuration. Thus, Matrix recovery management does not alter the IP address in the NIC in the OS to match that. You must modify the IP address of the OS manually. For example, if the IP address of the VM at the primary site is configured to 10.10.10.1, but the DR site subnet is 12.12.12.0, Matrix IO allocates 12.12.12.X IP address to the replica service. But when the OS boots, you must manually configure the IP address of the NIC to 12.12.12.X, as its primary IP address still refers to 10.10.10.X. DR protection for IO services 27 3 Testing and failover operations This chapter describes Recovery Group testing, planned failovers, and unplanned failovers using the Matrix recovery management Activate and Deactivate operations. Testing Recovery Groups There are two ways to test Recovery Groups: • Using Maintenance Mode to test individual Recovery Groups. NOTE: VC-hosted logical servers imported at the recovery site recovery group must be activated and then deactivated using Visualization Manager before maintenance mode can be disabled in Matrix recovery management. • Performing a planned failover to test all Recovery Groups. See “Planned failover” (page 29) for more information. This section focuses on using Maintenance Mode to test individual Recovery Groups. For testing purposes, you can manually place a Recovery Group into Maintenance Mode and then activate the deactivated logical servers or inactive services belonging to that Recovery Group using the HP Matrix Operating Environment. Maintenance Mode temporarily stops Matrix recovery management from managing DR Protected logical servers or DR Protected IO services. One use of Maintenance Mode is to perform a failover rehearsal. By setting Maintenance Mode on a Recovery Group, all of the logical servers and IO services in that Recovery Group can be activated or deactivated from the Matrix infrastructure service user interface. Once you are satisfied with the failover rehearsal, a Recovery Group and its corresponding logical servers or IO services can be brought back under the control of Matrix recovery management by disabling Maintenance Mode on that Recovery Group. At the Remote Site, the logical servers being managed by Matrix recovery management are known as recovery logical servers. Normally, a recovery logical server cannot be activated except by invoking an Activate operation from the Matrix recovery management user interface at the Remote Site. To use Maintenance Mode to test a Recovery Group: 1. At the Local Site, use the Logical Servers Deactivate operation in the Tools menu of the Visualization tab in the HP Matrix Operating Environment to gracefully shut down the logical servers in the Recovery Group. For IO services, use the Deactivate Servers IO service operation in the Matrix infrastructure service to shut down the IO services in the Recovery Group. NOTE: If the Matrix recovery management configuration includes Hyper-V logical servers or IO services, bring the cluster disk resource used for logical server or IO service storage offline prior to failing over storage to the remote site. If you did not bring the CSV offline, see “Recover the CSV from online pending state” (page 72) for how to recover. 2. 3. 28 At the Remote Site, use the appropriate storage management tools, (for example, HP P6000 Continuous Access Software), to fail over the storage corresponding to the Recovery Group to the Remote Site. If there are logical servers or IO services in the Storage Replication Group that will be activated on VM hosts during the test: a. Rescan storage using VM host management tools, for example, VMware Virtual Center, to ensure that the VM host recognizes the failed over storage. b. Refresh Virtual Machine resources using the Logical Servers Refresh operation in the Tools menu of the Visualization tab in Matrix OE visualization. Testing and failover operations 4. Place the Recovery Group into Maintenance Mode at the Remote Site using the Enable Maintenance Mode button in the Matrix recovery management Recovery Groups tab. For logical servers, use the Logical Servers Activate operation in the Tools menu of the Visualization tab in Matrix OE visualization to activate the logical servers in the Recovery Group at the Remote Site. Depending on the type of logical server, the activation may be on VC blades, VM hosts, or both. For IO services, select the Activate Servers service operation in the Matrix infrastructure service to activate the IO services in the Recovery group. When the test is complete, gracefully shut down the operating systems, then deactivate the logical servers or IO services. NOTE: If the Matrix recovery management configuration includes Hyper-V logical servers or IO services, bring the cluster disk resource used for logical server or IO service storage offline prior to failing over storage. If you did not bring the CSV offline, see “Recover the CSV from online pending state” (page 72) for how to recover. 5. 6. At the Remote Site, take the Recovery Group out of Maintenance Mode by using the Disable Maintenance Mode button in the Matrix recovery management Recovery Groups tab. At the Local Site, repeat the storage failover, rescan and refresh (if needed) sequences, then activate the logical servers in the Recovery Group using Matrix OE visualization. When you import a Matrix recovery management configuration at the Remote Site, all Recovery Groups that are imported have Maintenance Mode enabled by default, so they can be tested. Similarly, a Recovery Group created at the Remote Site has Maintenance Mode enabled by default. After testing is completed for all imported Recovery Groups at the Remote Site, deactivate the recovery logical servers or replica IO services belonging to each Recovery Group , then disable Maintenance Mode on each Recovery Group. Recovery Groups that have Maintenance Mode enabled are not included in an Activate operation. HP recommends that you carefully test Recovery Group failover at the Remote Site prior to disabling Maintenance Mode on imported Recovery Groups. Failover operations This section explains the difference between planned and unplanned failovers, and provides the procedure to follow in either case. A note about Microsoft Hyper-V Server failover: Matrix recovery management only supports VM failover between equivalent versions of Microsoft Hyper-V Server, or from earlier versions to current versions. Failover from a current version to an earlier version, such as failover from Hyper-V Server 2012 to Hyper-V Server 2008 R2, is not supported. Planned failover A planned failover typically involves an expected outage. For example, a planned failover might be necessary to perform scheduled maintenance, or to react to a severe weather forecast. The following section describes the steps necessary to perform a planned failover using Matrix recovery management. A planned failover includes a series of steps performed at both the Local site and the Remote site. A planned failover includes a series of steps performed first at the site you plan to shut down, and then at the other site in the Matrix recovery management configuration. At the site you want to shut down: Failover operations 29 1. 2. Shut down the applications and operating system on each Matrix recovery management DR Protected logical server and each server associated with DR Protected IO services. Click on the Deactivate... button and the Deactivate Recovery Groups at the Local Site window will appear. For more information about the Recovery Groups contained in a Recovery Group Set, select the Recovery Group Set and click View Recovery Group. A window will appear displaying the following parameters for each Recovery Group in the Recovery Group Set: 3. 4. • Name • Status • Type • Preferred Site • Secondary Site • Storage Replication Group Name • Start Order • Power-Up Delay Click the check-box on the left side of the banner of the Deactivate Recovery Groups at the Local Site window to select all of the Recovery Group Sets at the Local Site for deactivation. Click Deactivate Recovery Groups to start the deactivation operation. A window will appear asking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where you can monitor the progress of the deactivation Job. NOTE: If the Matrix recovery management configuration includes Hyper-V logical servers or IO services, bring the cluster disk resource used for logical server or IO service storage offline prior to activating operation at the remote site. If you did not take the CSV offline, see “Recover the CSV from online pending state” (page 72) for how to recover. At the site you want to fail over to: 1. Ensure that enough resources are available to run the logical servers that will be activated at this site. 2. Click on the Activate... button and the Activate Recovery Groups at the Local Site window will appear. For more information about the Recovery Groups contained in a Recovery Group Set, select the Recovery Group Set and click View Recovery Group. A window will appear displaying the following parameters for each Recovery Group in the Recovery Group Set: 30 • Name • Status • Type • Preferred Site • Secondary Site • Storage Replication Group Name • Start Order • Power-Up Delay Testing and failover operations 3. 4. Select each Recovery Group Set that you want to activate or click the check-box on the left side of the banner of the Activate Recovery Groups at the Local Site window to select all of the Recovery Group Sets for activation. Click Activate Recovery Groups to start the activation operation. A window will appear asking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where you can monitor the progress of the activation Job. NOTE: A successful Activate or Deactivate operation ensures that all of the Recovery Groups within a Recovery Group Set are in the same state (activated or deactivated). However, certain operations (for example, a Recovery Group edit to change site preference) may result in some Recovery Groups within a Recovery Group Set being activated and others being deactivated. You must run an Activate or Deactivate operation on a Recovery Group Set to ensure that all of the Recovery Groups in that Recovery Group Set are in the same state ( all activated or all deactivated). Unplanned failover An unplanned failover typically involves the occurrence of a site-wide event, without prior warning. This event may be a regional disaster (earthquake, massive flood), or a local problem (power loss or water main leak in the data center). An unplanned failover includes a series of steps performed at both sites in the Matrix recovery management configuration. The following procedure assumes that the CMS and managed resources running DR Protected logical servers survived an unplanned local event (for example, a power loss). If the event is more severe, resulting in the permanent loss of the CMS or managed resources, reconstruction of the site may be necessary. At the site where the site-wide event occurred: 1. Ensure that the DR Protected logical servers are no longer running in order to prevent a split-brain situation. As long as the DR Protected logical servers have stopped running, Matrix recovery management will prevent them from automatically powering up when power is restored. Matrix recovery management is able to prevent split-brain from occurring during an unplanned failover by regulating the auto-power configuration of managed nodes (whether virtual or physical) that are assigned to DR Protected logical servers so they do not automatically power-up after an outage. If, for example, a site loses power and site failover is invoked, the site where the power outage occurred will not resume running the DR Protected logical servers when power is restored. The managed nodes (whether VC blades or virtual machines) assigned to DR Protected logical servers stay powered down (and resources remain unassigned) until an Activate operation is invoked. At the failover destination (recovery) site: 1. Ensure that enough resources are available to run the recovery logical servers. 2. From the Matrix recovery management Sites tab, click the Activate... button and the Activate Recovery Groups at the Local Site window will appear. For more information about the Recovery Groups contained in a Recovery Group Set, select the Recovery Group Set and click View Recovery Group. A window will appear displaying the following parameters for each Recovery Group in the Recovery Group Set including: • Name • Status • Type • Preferred Site • Secondary Site • Storage Replication Group Name Failover operations 31 3. 4. • Start Order • Power-Up Delay Select each Recovery Group Set that you want to activate at the recovery site. The objective is for all Recovery Group Sets that were previously activated at the site where the site-wide event occurred to be activated at the recovery site. Click Activate Recovery Groups to start the activation operation. A window will appear asking if it is OK to proceed. Click OK and you will be directed to the Jobs tab where you can monitor the progress of the activation Job. NOTE: • If the Matrix recovery management configuration has been changed since the failover occurred (for example, a new Recovery Group was created), the sites must be brought into sync by making appropriate configuration changes. The Matrix recovery management Site configuration export and import operations can be used for this purpose. • A successful Activate or Deactivate operation ensures that all of the Recovery Groups within a Recovery Group Set are in the same state (enabled or disabled). However, certain operations (for example, a Recovery Group edit to change site preference) may result in some Recovery Groups within a Recovery Group Set being enabled and others being disabled. You must run an Activate or Deactivate operation on a Recovery Group Set to ensure that all Recovery Groups in that Recovery Group Set are in the same state (enabled or disabled). Target selection and parallelism during an activation operation The HP Matrix OE logical server management (logical server management) component in the HP Matrix Operating Environment supports the concept of targets that are most suitable for activating a logical server based on various criteria, for example, an application may need to run on VC-hosted logical servers only, to meet a performance requirement. DR Protected logical servers that can run on both physical and virtual targets (cross technology logical servers) will be placed on the target type specified as preferred in the site configuration (P for physical or V for virtual), based on availability. If the preferred target type is not available, Matrix recovery management will ignore the target type preference and place cross technology logical servers on available supported targets. Matrix OE logical server management allows logical servers to be activated in parallel, exploiting parallelism that is available in the managed infrastructure. Matrix recovery management exploits the parallelism of activation available from logical server management when performing an Activate operation, to reduce failover time. Two user accessible settings are available to influence Matrix recovery management behavior in this area: • Use Recovery Group Start Order values to determine the workloads that will be brought up first during the failover process. If there are workloads that you want to bring up first during the failover process, the Recovery Group Start Order values for the associated Recovery Groups can be set lower than the values you set for workloads that can be brought up later in the failover process. Matrix recovery management ensures that logical servers in a Recovery Group with a lower Recovery Group Start Order value are activated before logical servers in a Recovery Group with a higher Recovery Group Start Order value are activated. • 32 Use the Recovery Group Power-Up Delay parameter to ensure that the logical servers in a Recovery Group boot in a staggered fashion during the failover process. The Recovery Group Power-Up Delay parameter sets a minimum delay between the time when one logical server in the Recovery Group begins its boot process and the time when the next logical server in the Recovery Group begins its boot process. Testing and failover operations 4 Dynamic workload movement with CloudSystem Matrix This chapter explains how you can configure cross-technology logical servers that can be managed with Matrix recovery management. The HP Matrix Operating Environment facilitates the fluid movement of workloads between dissimilar servers within a site and across sites. Workloads can be moved between physical servers and virtual machines and between dissimilar physical servers. A major trend today in IT data center management is the push toward greater efficiency in the use of computing, network, and storage resources in the datacenter by treating them as a shared pool from which the resource requirements of various applications, departments, and organizations are met. Central to this concept of a converged infrastructure is the ability to rapidly and automatically create, move, and remove workloads on demand. In a typical converged infrastructure implementation, a customer may use HP CloudSystem Matrix to run the workloads and the HP Matrix Operating Environment running on a Central Management Server (CMS) to create, move, and remove the workloads as needed. The workload, which includes the operating system (OS) that the user application runs on, can run directly on a blade or it can run in a virtual machine managed by a hypervisor running on the blade, for example, VMware ESX. The blades may also have different hardware configurations or contain different versions of hardware and firmware. The capabilities of the HP Matrix Operating Environment discussed in this chapter allow fluid movement of workloads in this type of heterogeneous environment. These capabilities include: • Tools that allow the workload OS to be prepared as a portable system image that can run in different server environments. • Fine-grained user control over the set of specific physical servers and virtual machine hosts that the HP Matrix Operating Environment can run the workload on. The fluid, two-way movement of workloads across dissimilar servers described in this chapter is different from the movement enabled by traditional migration tools. Those tools are oriented towards enabling a one-way, permanent or semi-permanent migration, between physical and virtual or between dissimilar physical servers. The movement typically requires manual intervention and a relatively long period of time to complete. The importance of the ability to fluidly move a workload from a physical server to a virtual machine and back can be understood from the following examples: • You want to move your online workload running on a physical server during daily off-peak hours to a virtual machine host, to free up the physical server to run a batch workload. When the off-peak period is over, the batch workload is retired and the online workload is moved back to its original execution environment. During the time the online workload is “parked” on a virtual machine host, it has minimal resource requirements; hence it has minimal impact on other workloads that may be running on that host. Because this pattern repeats daily, the physical to virtual and virtual to physical moves must be achieved quickly (in minutes, rather than hours) and automatically. • You have two data centers located at two different sites. The production workloads run on physical servers at one site and are configured to be failed over to the other (recovery) site, in case of a disaster. The recovery site is equipped with a set of servers configured as virtual machine hosts. In this use case, planned or unplanned failovers require physical to virtual and virtual to physical moves across sites. The configuration of the recovery site as a set of virtual machine hosts may be driven by the needs of test and development activities that are carried out at that site. Or, it may be driven by a need to reduce the cost of disaster recovery by running the workloads on virtual machines hosted by a smaller set of servers. Recovery time objectives require that the moves be achieved quickly and automatically, as in the previous example. 33 Capabilities and limitations Using the tools and procedures described in this chapter you can: • Configure and manage a logical server that can perform physical to virtual cross-technology movements within the datacenter. • Configure and manage a DR Protected logical server that can be failed over across data centers in a cross-technology movement. The following limitations should be noted: 34 • Configuration of a cross-technology logical server requires additional steps (however, no additional steps are required at the time of the move, within or across sites). • Virtual machines running Windows 2008 or later are supported. Virtual machines must be configured to emulate either the LSI Logic Parallel or LSI Logic SAS storage type. • Because DR Protection for IO services is supported for VM based IO services only, cross-technology movement of IO services workloads across data centers does not apply. • VMware ESX guest tools are not automatically installed. • There is no explicit support NPIV. • Preparation of the portable system image so it can run on both physical and virtual servers is done from an OS installed on a physical server. You cannot prepare the portable system image from an OS installed on a virtual machine. • Virtual machines must be configured to use RDM Fibre Channel SAN storage presented to the VM host for boot and data. This is the same storage that a logical server uses when running on a physical server. • Each boot and data LUN must use the same LUN number across the physical and virtual targets for a logical server as illustrated in Figure 6 (page 35). Dynamic workload movement with CloudSystem Matrix Figure 6 Same LUN number across physical and virtual targets • The target WWN values used to present the Logical Unit must be the same across virtual and physical targets. NOTE: The recovery logical server that provides DR protection at the Remote Site has its own set of target WWN/LUN values that differ from the target WWN/LUN values for the logical server at the Local Site. • The network name used by an ESX Host must match the network name used in the Virtual Connect Enterprise Manager configuration, as displayed in Figure 7 (page 36) and Figure 8 (page 36). Capabilities and limitations 35 Figure 7 ESX host network name Figure 8 Virtual Connect Enterprise Manager network name • • 36 When moving a logical server between physical and virtual servers within a site, the following server IDs are not preserved: ◦ Network MAC addresses ◦ Server/Initiator WWNs (On a virtual machine, the storage adapter is a virtual SCSI controller.) ◦ Logical Serial Number ◦ Logical UUID In a DR configuration, the site that you configure first must have both physical servers and virtual machine hosts available so a logical server can be configured to run and be tested on Dynamic workload movement with CloudSystem Matrix both types of servers. The recovery site can have a physical/virtual combination also, or have only virtual machine hosts. Supported platforms The procedures for enabling movement between physical and virtual servers described in this chapter apply to physical servers, hypervisors, and workload operating systems supported by Matrix recovery management. For more information, see the HP Insight Management 7.3 Update 1 Support Matrix at Enterprise Information Library. • For supported hypervisors, see VMware Hypervisor versions specified as supported on managed systems by the Matrix Operating Environment. • For supported workload operating systems, see Windows 2008 versions specified as supported on managed systems by the HP Matrix Operating Environment. The procedures for enabling movement between physical and virtual servers are not supported on Integrity managed nodes. The procedures for enabling movement across different physical servers documented in this chapter are supported for managed systems specified as supported by the HP Matrix Operating Environment, with the following restriction: • Matrix recovery management, the component of the HP Matrix Operating Environment that provides disaster recovery across sites, does not support Integrity managed nodes. A physical server target configured for cross-technology movements must be an HP c-Class blade with HP Virtual Connect. Overview of physical to virtual cross-technology configuration This section provides an outline of the steps involved in configuring cross-technology logical servers for movement between physical and virtual targets, and for movement between dissimilar physical servers. Configuring logical servers for movement between physical and virtual targets 1. Prepare a logical server with a portable image. Start with a logical server configured to run on a physical server, and prepare its system image for movement between physical and virtual servers. a. Storage configuration The Portable Images Storage Assistant (PISA) tool prepares the storage configuration of the server image so it can be booted in both physical and virtual environments. PISA is part of the HP Insight Control server migration product on the HP Insight Management DVD. The executable and README are in the \PI\PISA folder, where is the directory where Insight Control server migration is installed (the default install directory is C:\Program Files\HP\Insight Control server migration). i. Copy the executable hppisa.exe (under PI\PISA) to the physical server where the image is currently running. ii. In the command-line window, type: > hppisa –e For more information, see “Portable Images Storage Assistant (PISA)” (page 40). b. Network configuration Portable Images Network Tool (PINT) prepares the image to execute on targets with different network interface configurations and MAC addresses. It ensures that the static network configuration from the source server is successfully transferred to destination server network interfaces despite the differing environment. The executables and README are in the folder \PI\PINT. Overview of physical to virtual cross-technology configuration 37 i. ii. Copy the executable cp011231.exe to the physical server where the image is currently running. Run cp011231.exe to install PINT and start the PINT service. For more information, see “Configuring and managing portable OS images” (page 40). 2. Create a portability group that includes all potential physical and VM host targets. This step sets up the portability group that defines the list of potential targets for the logical server. The group should include both physical servers and VM hosts as targets. See Figure 9 (page 38). Figure 9 Creating a portability group For more information, see “Portability groups” (page 41). 3. Configure the logical server for activation on both physical and VM host targets. Modify the logical server configuration as follows: • In the Create logical server: identity screen, set the portability group of the logical server to the portability group created in Step 2. • In the Create logical server: storage screen, select a VM datastore (this datastore will be used to store VM configuration information). For more information, see “Defining cross-technology logical servers” (page 43). • 4. Present the boot/data storage to the VM hosts in the portability group using the same LUN and WWN values. For more information, see “Storage definition” (page 45). Perform physical to virtual and virtual to physical movements to verify OS configuration. Move the logical server from physical to virtual and back again. After each move, ensure that the logical server has booted successfully and the static network configurations have been applied as expected to the network interfaces present in the new environment. For more information, see “Moving between technologies” (page 46). 38 Dynamic workload movement with CloudSystem Matrix NOTE: When the logical server is first moved to a virtual machine, you may want to add additional tools to the server, for example, VMware tools. In the HP Matrix Operating Environment, the VM configuration created does not include a virtual CD/DVD drive. You can use the VM management console to modify the VM configuration to include a virtual CD/DVD drive. 5. Configure inter-site movement between physical and virtual targets (disaster recovery use case). In this step, Matrix recovery management capabilities are used to set up inter-site movement between physical and virtual targets. a. At the Local Site, create an array replication group containing the LUNs used by the logical server for boot/data and the LUN containing the VM datastore as illustrated in Figure 10 (page 39). The VM datastore should be used exclusively to store the VM configuration of the logical server. Figure 10 Creating an array replication group b. c. d. e. f. g. h. i. At the Local Site, create a Recovery Group containing the logical server. Export the Matrix recovery management configuration to a file. Deactivate the logical server at the Local Site and failover the array Replication Group to the Remote Site. Perform VM host rescan and Matrix OE visualization refresh procedures to ensure that the VM configuration datastore is accessible to the logical server configuration. Create a portability group at the Local Site. The portability group can contain physical servers, VMs, or a combination of both. Create the recovery logical server with the same configuration values used at the Local Site, but adjusted to point to the remote storage LUN/WWN values. Do not activate the logical server at this time. Using the exported configuration from the Local Site, import the Recovery Group that was created at the Local Site to contain the recovery logical server. The Recovery Group is in Maintenance Mode. Activate the logical server on a VM host. If one or more physical servers are available in the portability group, perform virtual to physical and physical to virtual movements. At each stage, check for successful boot and confirm correct network configuration, as in step 4. Deactivate the logical server and clear Maintenance Mode. Fail the array replication group back to the Local Site. At the Local Site, perform rescan and refresh procedures, and activate the logical server. Overview of physical to virtual cross-technology configuration 39 NOTE: The Matrix recovery management Site configuration can be set up to preferentially activate the logical server on a physical server at one site and a VM host at the other site. For more information, see “Setting a failover target type preference” (page 48). Configuring logical servers for movement between dissimilar physical servers The HP Matrix Operating Environment provides the ability to fine tune the list of failover targets that are considered most suitable for a DR Protected logical server to be activated on. The ability to modify target attributes is useful to ensure a successful failover. Target attributes are included in the data transferred by the Matrix recovery management export/import sequence. An expansion in the target list at the exporting site is reflected at the importing site. For more information, see “Target attributes” (page 47). Configuring and managing portable OS images Mobility of server workloads is hampered by the fact that most operating systems are configured at install time for the specific platform they are installed on. Examples include: • Only the device drivers necessary for the target platform are installed and configured; attempting to boot the same OS instance on a different server may result in device errors or system failures. • Configuration settings, such as IP addresses or storage identifiers, may be bound to specific devices whose names may change. For example, the same subnet may be attached to the first NIC port on one server, while it is attached to the third NIC port on another. The HP Matrix Operating Environment supports mechanisms to prepare a server image (logical server) so it continues to work when moved to a different physical or virtual server. The following areas are key to achieving this objective: • Driver installation • HBA configuration • NIC configuration The tools developed for this purpose are as follows: • Portable Images Storage Assistant (PISA) • Portable Images Network Tool (PINT) Portable Images Storage Assistant (PISA) When Windows is installed on a SAN LUN attached to a server, it installs a driver that is specific to the HBA controller on that server. If that LUN is subsequently reassigned to a virtual machine on a server running VMware ESX, ESX presents only SCSI direct-attached devices to the virtual machine. Because Windows is still configured to use the HBA controller for the original server it is unable to start up. PISA enables the appropriate Windows drivers so Windows can start in the virtual machine. PISA requires that the virtual machine is configured to use the Raw Device Mapping feature in ESX to configure a LUN on a SAN as a disk drive available to the virtual machine. The virtual machine must also be configured to emulate either the LSI Logic Parallel or LSI Logic SAS storage type. In all versions of Windows, the driver needed to properly operate the virtual version of the LSI controller is installed, but disabled. PISA is used to enable LSI support in the Windows image. PISA is a simple command-line tool that accepts only a few command-line options. It needs to be executed only once after Windows has been installed on a physical server. The changes it makes are persistent and do not need to be repeated or reversed. However, repeatedly running PISA has no negative impact. PISA can also be used to disable the driver used by the virtual machine. 40 Dynamic workload movement with CloudSystem Matrix The command-line interface for PISA is described below. The options are mutually exclusive. PISA runs on supported versions of Windows only, and it requires that the user be a member of the Administrator user group. Usage: hppisa -h, -?, -help Show this information -e, -enable Enable the LSI driver -d, -disable Disable the LSI driver After these changes are made, the OS image can be moved back and forth between physical servers and virtual machines. For more information on PISA, including installation, supported operating systems and operating instructions, see the readme available on the HP Insight Control DVD at C:\Program Files\ HP\Insight Control server migration\PI\PISA. Portable Images Network Tool (PINT) PINT is used to resolve networking issues when you move an OS image from one physical server to another, or from a physical server to a VMware ESX virtual machine. However, when an OS is booted after a VC-hosted logical server activation at the DR site, PINT might not assign the correct IP address to the intended NIC. If this happens, manually configure the NIC IP address. PINT maintains a network configuration file where it gathers information on each network interface and its configuration on a server. PINT remains in a suspended state, running only when it receives one of the following events: • IP configuration change event PINT considers any changes made while the server is up and running to be intentional changes made by the user. PINT records the changes and updates its configuration file. • Stop event If the user stops the PINT service PINT receives an event that lets it know to shutdown. • User command event If a user makes any changes through the PINT command-line PINT is notified and acts accordingly. NOTE: If a NIC in the destination server requires a different set of drivers than those on the source server, you must install the new drivers before using PINT on the destination server. For more information on PINT, including installation and operating instructions, see the Portable Images Network Tool (PINT) Windows readme or the Portable Images Network Tool (PINT) Linux readme available in the PINT directory on the HP Insight Control DVD at C:\Program Files\ HP\Insight Control server migration\PI\PINT. Configuring and managing cross-technology logical servers This section explains configuration tasks and management of cross-technology logical servers in an HP Matrix Operating Environment. Portability groups When you create a logical server, you must specify the portability of the logical server. You do this by selecting a portability group from within the identity page of the logical server configuration wizard. After a logical server is associated with a particular portability group, it can be moved to any target system (HP Virtual Connect physical server or virtual machine hypervisor) within that portability group. Logical server resource constraints, such as CPU/memory requirements and network/SAN connectivity, are evaluated solely within the context of the portability group that the logical server is associated with. Portability groups come in two classes: Default and User Defined. Configuring and managing cross-technology logical servers 41 The HP Matrix Operating Environment provides default portability groups depending on the resources found within your data center. The Default portability groups include: • ESX—All ESX Hypervisors • HYPERV—All Hyper-V Hypervisors • Each Virtual Connect Domain Group—Each VCDG has its own Default portability group. You can also create User Defined portability groups that extend the portability of a logical server to unlike technologies. For example, moving logical servers between a Virtual Connect physical server and a VMware ESX virtual machine host. User Defined portability groups are defined by selecting Modify→Logical Server Portability Groups in Matrix OE visualization as illustrated in Figure 11 (page 42). Figure 11 Modifying logical server portability groups If you have selected one or more targets in Matrix OE visualization, they are presented as potential targets. Otherwise, all resources are presented. Figure 12 (page 43) provides an example of the Modify Portability Group screen. 42 Dynamic workload movement with CloudSystem Matrix Figure 12 Selecting group members and targets Provide a name and optional description for the portability group. The name will be used for defining logical servers. The set of Group Types is selected automatically based on the targets inserted into the portability group. Valid combinations of targets include: • A single Virtual Connect Domain Group (VCDG) • A set of ESX Hypervisors • A set of Hyper-V Hypervisors • A set consisting of a single VCDG plus a set of ESX Hypervisors. Defining cross-technology logical servers To define a cross-technology logical server, you must place a logical server into a portability group, and then define the storage for that logical server. Placing a logical server into a portability group Placing a logical server into a portability group is accomplished on the logical server identity page. This can be done during the creation of the logical server or in a subsequent logical server modification. Select from the Portability Group list as shown in Figure 13 (page 44). This list includes both the Default portability groups as well as any User Defined portability groups. Configuring and managing cross-technology logical servers 43 Figure 13 Selecting a portability group To view the portability group for any logical server, click the View movable logical server details icon in Matrix OE visualization as illustrated in Figure 14 (page 44). Figure 14 View movable logical server details icon The details for this logical server are displayed as illustrated in Figure 15 (page 44). Figure 15 View logical server details 44 Dynamic workload movement with CloudSystem Matrix Logical servers can be made portable through techniques described in “Portability groups” (page 41). NOTE: You must determine whether the provisioned operating system within a logical server performs as desired on a variety of platforms. If a logical server has never been active on a platform type, the HP Matrix Operating Environment shows a warning for each target of that type in the Target Selection page during moves and activations. You must determine whether the target is valid. Storage definition Storage can be defined through Storage Pool Entries or Storage Entries tied directly to a logical server. The storage for cross-technology logical servers must be SAN-based. This approach uses the normal SAN-boot approach within Virtual Connect and leverages ESX RDM technology which presents boot and data LUNs directly to the virtual machine. For examples, see Figure 16 (page 45) and Figure 17 (page 45). Figure 16 Storage Pool Entries Figure 17 Storage Entries Configuring and managing cross-technology logical servers 45 When defining storage for a portable logical server, you must select SAN Storage Entry. For flexibility and movement between underlying technology types, storage must be presented to the WWNs tied to the Virtual Connect server profile, and storage must also be presented to any ESX VM hosts that are potential targets for the logical server. Storage Pool Entries must be created within the same portability group associated with the logical servers that will use the storage as illustrated in Figure 18 (page 46). Figure 18 Managing Storage Pools Moving between technologies Activation and movement of cross-technology logical servers is accomplished in the same way as with standard logical servers. However, the Unlike Move operation is used for cross-technology logical servers when an Activate or Move operation is about to be performed on a server with a different underlying technology from its previous target host. An Unlike Move operation is illustrated in Figure 19 (page 46). Figure 19 Assigning logical servers to target hosts 46 Dynamic workload movement with CloudSystem Matrix Targets for a logical server are selected from that logical server's portability group. The portability group members are then further filtered based on resource availability, including CPU and memory resources as well as network and SAN connectivity. NOTE: Networks in Virtual Connect must be named identically to their corresponding networks (port groups) on ESX Hypervisors. Differences in names prevent the Unlike Move operation from identifying networks with similar connectivity. Target attributes You can track where a logical server has been successfully activated or moved by using logical server target attributes. Target attributes provide a greater number of “most suitable” targets where you can activate or move a logical server. To view or modify target attributes on a logical server, select the logical server and then click Modify→Logical Server Target Attributes→Manage as illustrated in Figure 20 (page 47). Figure 20 Modifying logical server target attributes Types of targets can be selected and added or removed from the logical server's target attributes. Selecting a server from the list below and clicking Add adds that type of server with associated resources to the list of “most suitable” targets for the logical server. Selecting a type of server from the Target Attributes Available to Remove list and clicking Remove causes the type of server specified to no longer be listed as “most suitable.” Figure 21 (page 47) illustrates the screen where you can manage logical server target attributes. Figure 21 Managing logical server target attributes Configuring and managing cross-technology logical servers 47 Moving between blade types For logical servers with target attributes, the logical server management software can identify more possible targets when moving or activating a server. As with all cross-technology logical servers, you must ensure that the logical server can function appropriately on various platforms. If a particular target is proven to be unsuitable, it is easy to remove that type of target to more accurately describe the logical server's portability. Managing DR Protected cross-technology logical servers in a Matrix recovery management configuration This section explains how to specify failover target type preferences for DR Protected cross-technology logical servers in a Matrix recovery management configuration. Setting a failover target type preference During a site failover, for every logical server that has been configured with disaster protection, Matrix recovery management activates a similarly configured peer logical server at the recovery site. For this purpose, Matrix recovery management interacts with the Matrix OE logical server management capability to determine a list of appropriate available targets and chooses the most suitable target to activate the logical server. For a cross-technology logical server, the appropriate list of available targets may include both physical servers and VM hosts. Matrix recovery management includes a Target type preferred setting in the Sites configuration tab, where you can specify the type of target preferred for each site, as illustrated in Figure 22 (page 48). Figure 22 Matrix recovery management Sites configuration screen 48 Dynamic workload movement with CloudSystem Matrix You must specify the target type preferred for all sites on the CMS at each site: • If you specify Virtual as the target type preferred for a site, all cross-technology logical servers whose Recovery Groups prefer that site are activated on VM hosts during an Activate operation at that site. A physical server is chosen only if no VM hosts are available. • If you specify Physical as the target type preferred for a site, all cross-technology logical servers whose Recovery Groups prefer that site are activated on physical servers during an Activate operation at that site. A VM host is chosen only if no physical servers are available. By specifying appropriate target preferences for sites, you can set up a configuration where DR Protected cross-technology logical servers run on physical servers at one site and on VM hosts at the other site. For example, you might choose to have DR Protected cross-technology logical servers run on physical servers at the preferred site, but run on VM hosts when they failover to the secondary site. Managing DR Protected cross-technology logical servers in a Matrix recovery management configuration 49 5 Issues, limitations, and suggested actions This chapter lists issues and limitations for this release, categorized as follows: Limitations Limitations of the implemented functions and features of this release Major issues Issues that may significantly affect functionality and usability in this release Minor issues Issues that may be noticeable but do not have a significant impact on functionality or usability Limitations • Only IO services that include virtual servers and on-premise (not cloud) resources are supported. • A Recovery Group can contain logical servers only, or IO services only, but not a mix of logical servers and IO services. • Replica IO services cannot be flexed (add disk, add server, remove server). • Replica IO services cannot be imported to create primary services in a Matrix recovery management configuration. Hyper-V support limitation for bidirectional configuration For bidirectional failover configurations, the logical servers and IO services configured in Recovery Groups with the Local Site as the Preferred Site are required to have read-write access to the associated storage on the Local Site. The logical servers and IO services configured in Recovery Groups with the Remote Site as the Preferred Site may have read-only access to the associated storage on the Local Site. Microsoft requires that all of the storage (cluster shared volume disks or cluster disks) in the clusters where the logical servers will be activated must have read-write access. Suggested action Microsoft has provided a hotfix. See the Microsoft Knowledge Base article and download the hotfix at http://support.microsoft.com/?id=2720218. No automatic synchronization of configuration between sites The configuration of Matrix recovery management at two separate sites is not automatically synchronized, but Matrix recovery management provides configuration export and import features that simplify this task. Suggested actions The Matrix recovery management online help guides you through the export and import operations. Matrix recovery management job information is not preserved in certain scenarios Information about previously executed Matrix recovery management Jobs is not preserved when the Matrix recovery management configuration is restored using the HP Insight mxsync utility. Minor issues Firefox browser cannot be used for site export operations Firefox browser cannot be used to perform Matrix recovery management site configuration export operations, due to a bug in Adobe Flash Player 10.3. For more information, see Bug 2980517 on the Adobe web site. Suggested actions Use Internet Explorer 8 or later to perform Matrix recovery management site configuration export operations. 50 Issues, limitations, and suggested actions ESX configuration setting required for VMFS datastores of Matrix recovery management managed logical servers to be visible at Remote Site Under the following conditions, Matrix recovery management requires a specific ESX configuration setting to retain the signature of a VMFS datastore so it will be visible at the Remote Site: • You have asymmetric HP P6000 Continuous Access Software array models at the Local and Remote Site. • You are using HP P9000 Continuous Access Software storage arrays at the Local and Remote Site. • You are using HP 3PAR storage systems. Suggested actions • Login to all the ESX hosts and execute the following command: # esxcfg-advcfg –s 0 /LVM/DisallowSnapshotLUN • For ESX hosts, to mount the Local Site datastore with an existing signature, see “Mount a VMFS Datastore with an Existing Signature” in the ESX Configuration Guide Update 1, ESX 4.0, vCenter Server 4.0 available at http://pubs.vmware.com/vsphere-4-esxi-installable-vcenter/ index.jsp?topic=/com.vmware.vsphere.esxi_server_config.doc_41/esx_san_config/ managing_san_systems/c_mounting_vmfs_datastores_with_existing_signatures.html. For additional information, see the VMware Knowledge Base at http://kb.vmware.com/kb/ 1015986. Activation or deactivation job hangs If Matrix OE logical server management is not able to complete a task initiated by a Matrix recovery management Acitvate or Deactivate operation due to underlying stack issues, the Matrix recovery management operation hangs. Suggested actions: Restart the HP Logical Server Automation service from the CMS (Windows Administrative Tools→Services and Applications→Services) and perform the Matrix recovery management operation again. Identical configuration of logical servers between sites Local and recovery logical servers must be configured with identical parameters (except for the logical server name) using Matrix OE visualization. There is a potential for discrepancies, between the Local Site and the Remote Site, in the values of attributes that are not included in the Matrix recovery management user interface configuration screens. For example: • MAC address—For VC-hosted logical servers, the MAC address is assigned from Virtual Connect or Virtual Connect Enterprise Manager. Because disjointed address ranges must be used at the Local and Remote Sites, the MAC address for a Local Site logical server and it’s corresponding Recovery Site logical server are different. • HBA WWN—For a VC-hosted logical server, the HBA WWN address is assigned by Virtual Connect or Virtual Connect Enterprise Manager. Because disjointed address ranges must be used at the Local and Remote Sites, the HBA WWN address for a Local Site logical server and it’s corresponding Recovery Site logical server are different. • BIOS UUID—There is no supported mechanism to preserve the UUID for a VC-hosted logical server as it moves across sites (as there is when it moves within a site). • BIOS serial number—There is no supported mechanism to preserve the serial number of a VC-hosted logical server as it moves across sites (as there is when it moves within a site). • Array LUNs—On a VC-hosted logical server, the Windows or Linux OS must map presented LUNs to the volumes and file systems configured for the OS. For a VM-hosted logical server, Minor issues 51 the ESX OS on the host must map the presented LUNs to the VMFS used by the VM-hosted logical server. Although the operating systems have built-in mechanisms to do this, HP recommends as a best practice that you keep LUN numbers the same for corresponding disks across sites. Suggested actions Assess the impact of these discrepancies on any licensing arrangements in use for the operating system and applications running on DR Protected logical servers. One RAID Manager instance per HP P9000 Storage Management Server and One RAID Manager instance per HP P9000 device group Each HP P9000 Continuous Access Software Storage Management Server configured in Matrix recovery management has only one HP P9000 RAID Manager Software instance managing the HP P9000 device groups. Each HP P9000 Continuous Access Software Storage Replication Group configured in Matrix recovery management is managed by only one HP P9000 RAID Manager Software instance at each site. Suggested actions: There is no workaround for this issue. CLX/HP P9000 software must be installed on a separate Windows system A separate Windows system other than the Central Management Server must be configured with CLX/HP P9000 and compatible HP P9000 RAID Manager Software to manage the various HP P9000 device groups included in a Matrix recovery management configuration. Suggested actions There is no workaround for this issue. One active Matrix recovery management configuration operation at any point in time If multiple users attempt to run configuration operations in Matrix recovery management, only one operation succeeds. All other configuration operations receive an error message indicating that some other configuration operation is in progress. Suggested actions There is no workaround for this issue. Site delete operation in Matrix recovery management does not remove HP SIM tools If HP P9000 Continuous Access Storage Management Servers are configured and if a site delete operation is performed, the HP SIM tools used to manage HP P9000 storage replication on the local Storage Management Server are not deleted. Suggested actions The mxtool command can be run manually to remove the HP SIM tools. The names of the tools are Insight Recovery_Failover and Insight Recovery_Group validation. For example, if the name of the Storage Management Server is stgmgmtA.cup.hp.com, the names of the tools are STGMGMTA_CUP_HP_COM_Insight Recovery_Failover and STGMGMTA_CUP_HP_COM_Insight Recovery_Group validation. 52 Issues, limitations, and suggested actions 6 Troubleshooting This chapter provides troubleshooting information in the following categories: • “Configuration troubleshooting” (page 53) • “Configuration error messages” (page 55) • “Warning messages” (page 58) • “Matrix recovery management Job troubleshooting” (page 59) • “Failover error messages” (page 62) • “Matrix recovery management log files” (page 63) • “DR Protected IO serivces troubleshooting” (page 63) Configuration troubleshooting To troubleshoot Matrix recovery management configuration operations, take note of any error messages that appear, then review this section for relevant information. You can also view the mxdomainmgr log files for additional information. The following troubleshooting issues are addressed in this section: • Unable to add or edit Site information Possible causes include: • ◦ The local or remote CMS name provided is not valid (not a fully qualified domain name or locatable in the DNS). ◦ The local or remote CMS name does not include a fully qualified name associated with the local host. Unable to add or edit Storage Management Server information Possible causes include: ◦ The Storage Management Server is not discovered in the HP Matrix Operating Environment user interface. ◦ The credentials associated with the Storage Management Server in the HP Matrix Operating Environment user interface do not include the user name provided as part of a Storage Management Server configuration that has been added or edited. ◦ The HP Matrix Operating Environment user interface is unable to communicate with the Storage Management Server. Communication Failure with Storage Management Servers may result if Open SSH is not being installed or configured on the target system. • Unable to add or modify HP P6000 Storage Management Server Possible causes include: ◦ The CIMOM Server is not running on the Storage Management Server. ◦ The CIMOM server is configured to use a different port than the one specified for add or edit operations. ◦ The specified user does not have valid login credentials on the Storage Management Server with appropriate privileges. Configuration troubleshooting 53 • Unable to add or edit HP P6000 Storage Replication Group Possible causes include: ◦ • Matrix recovery management is unable to obtain Storage Replication Group information from Command View servers to validate the Storage Replication Group information provided by the user. Unable to add or edit HP P9000 Storage Replication Group Possible causes include: • ◦ The Storage Replication Group is not configured to be managed by the RAID manager instances. ◦ The RAID manager service is not running on the local Storage Management Server. ◦ The parameters entered for the Storage Replication Group (local array, remote array, serial number, and replication mode) do not match the Storage Replication Group information on the disk array. Unable to add or modify HP 3PAR Storage Management Server Possible causes include: • ◦ The password file for the HP 3PAR storage system is missing from the /storage/3par/ conf directory where Matrix recovery management is installed. ◦ The password for the HP 3PAR storage system has been changed since the password file was created. Unable to add or edit HP 3PAR Storage Replication Group Possible causes include: • ◦ The Storage Replication Group is not configured to be managed by HP 3PAR remote copy. ◦ The following information entered for the Storage Replication Group cannot be validated with the HP 3PAR storage system configuration: – Volume group name on the local HP 3PAR storage system – Volume group name on the remote HP 3PAR storage system – Local array remote copy group name – Remote array remote copy group name – Local HP 3PAR storage system serial number – Remote HP 3PAR storage system serial number – Replication mode Matrix recovery management logical server configuration is inconsistent with Matrix OE logical server management logical server configuration Possible causes include: 54 ◦ The HP Logical Server Automation service was not running when Recovery Groups were configured in Matrix recovery management. ◦ The logical server was being actively managed when the Matrix recovery management configuration was invoked (logical server modification, logical server activation, or logical server deactivation was in progress). Troubleshooting • No configuration operation can be run Possible causes include: • ◦ An Activate, Deactivate, or Import operation is in progress. ◦ Another configuration operation may be in progress Unable to import Storage Management Servers as part of an import operation Possible causes include: • ◦ The Storage Management Server was not discovered in the HP Matrix Operating Environment user interface. ◦ The credentials associated with the Storage Management Server do not include the user name specified in the Matrix recovery management configuration at the Local Site. ◦ The HP Matrix Operating Environment user interface is unable to communicate with the Storage Management Server specified in the Matrix recovery management configuration at the Local Site. Import operation failed Possible causes include: ◦ The import file is not valid. ◦ The HP Logical Server Automation service is not running. ◦ There were recovery logical servers in an active state at the time of the import. Configuration error messages This section lists Matrix recovery management configuration error messages. Error message Error: Matrix recovery management is being quiesced. Currently running operations will be allowed to complete. No configuration changes are allowed at this time. Cause The user attempts to change the configuration of Matrix recovery management when the system is being quiesced. Action Wait for Matrix recovery management to be unquiesced and retry the configuration change. Error message Error: Matrix recovery management is quiesced. No configuration changes are allowed at this time. Cause The user attempts to change the configuration of Matrix recovery management when the system has been quiesced. Action Wait for Matrix recovery management to be unquieced and retry the configuration change. Error message You are not authorized to access this page. Cause The user is not authorized to use Matrix recovery management , or the local CMS hostname cannot be resolved by the DNS. Action Check the HP SIM and Matrix recovery management log files for details, then contact the network administrator to add the local CMS hostname to the DNS. Configuration error messages 55 56 Error message Cannot verify the host name specified. Cause The hostname specified for the CMS for either the Local Site or the Remote Site is not locatable in the DNS. Action Verify that a valid DNS entry with a fully qualified domain name exists for each CMS. Error message Cannot create/edit the site information. Cause The hostname specified for the CMS does not include a fully qualified domain name associated with the local CMS. Action Ensure that the CMS for the Local Site has the fully qualified domain of the local host. Error message Storage Management Server not discovered by CMS. Discover server and retry. Cause Each Storage Management Server to be configured in Matrix recovery management must be discovered in the HP Matrix Operating Environment user interface with appropriate credentials specified. Action Ensure that the Storage Management Server is discovered in the HP Matrix Operating Environment user interface. If not, discover the server by using Options→Discovery in the HP Matrix Operating Environment user interface. Error message Error: Invalid storage manager username and/or domain name. Cause The login credentials for the server stored in the HP Matrix Operating Environment user interface do not include the user name specified as part of the Storage Management Server configuration operation in Matrix recovery management. Action For the server specified, ensure that the login credentials stored in the HP Matrix Operating Environment user interface include the credentials for the user name specified. Error message Error: Failed to add/modify EVA Storage Management Server. Credential for Storage Management Server does not exist. Check input and retry. Cause The hostname, port number, and user name specified cannot be validated. Action Ensure that the server identified by the hostname is an HP P6000 Command view server. Ensure that CIMOM on the Command View server is configured to use the port number specified and that the user name specified is a valid user on that server. Error message Unable to add/edit Storage Management Server Cause The server is not properly configured to be managed by the HP Matrix Operating Environment user interface. Action Ensure that Open SSH is installed and configured on the server, and ensure that the managed nodes and CMS are trusted. This can be achieved by running Configure→Configure or Repair Agents in the HP Matrix Operating Environment user interface. Error message Unable to add/edit EVA Storage Replication Group Cause Unable to obtain information about the Storage Replication Group from the HP P6000 Command View Server. Action Confirm that the Storage Replication Group exists on the arrays that were listed when the HP P6000 Storage Replication Group configuration was added or edited in Matrix recovery management. Ensure that the HP P6000 arrays are managed by HP P6000 Command View servers. 1. Confirm that the SIM-S component of HP P6000 Command View has been installed on the HP P6000 Command View server. For information on installing SMI-S, see the HP P6000 Command View Software Installation Guide available at http://h20000.www2.hp.com. Click on Manuals, Troubleshooting then go to Storage→Storage Software→Storage Device Management Software→HP P6000 Command View Software. 2. Confirm that the port number specified during Storage Management Server configuration in Matrix recovery management is the same as the WBEM port number configured on the HP P6000 Command View server (for example, 5989). For more information, see the “CIMOM” server configuration section in the HP P6000 Command View Software Installation Guide. 3. Refresh the HP P6000 Command View server CIM database by running the refresh command. For details on the Discover command, see the “Configuring HP SMI-S EVA to Discover HP Command View EVA Arrays” section in the HP P6000 Command View Software Installation Guide. 4. Ensure that the HP P6000 Command View Software service and CIMOM service are running. Start or restart these services if necessary. Error message Unable to add/edit XP Storage Replication Group Cause Matrix recovery management is unable to obtain information about the Storage Replication Group from the RAID manager, or the information obtained does not match the information provided when the HP P9000 Storage Replication Group configuration was added or edited in Matrix recovery management. Action Ensure that the RAID manager instance on the local Storage Management Server is running. From the install directory of the RAID Manager, run pairdisplay –g (where is the name of the Storage Replication Group being added or edited) to see if the Storage Replication Group is configured and the local array serial number, remote array serial number, and storage replication type matches the data provided when the HP P9000 Storage Replication Group configuration was added or edited in Matrix recovery management. Error message Unable to add CLX credentials for this HP 3PAR storage system. Cause The encrypted password file for the corresponding HP 3PAR storage system is incorrect or the password for the storage system has been changed and cannot be authenticated using the existing password file. Action Ensure the correct password file is present in the /storage/3par/conf directory where Matrix recovery management is installed for both the local and remote HP 3PAR storage systems. If necessary, regenerate the password files. Retry the Storage Management Server Add operation. Error message Error: Unable to locate CLX/3PAR install path. Check if CLX/3PAR is installed on the system. Cause The required software HP 3PAR Cluster Extension Software is not installed, or the CLI version does not match the one in the hp_ir.properties file. Action Install only the CLI component of the HP 3PAR Cluster Extension Software by using the custom install option, or update the OS_CLI_VERSION property in conf\hp_ir.properties where Matrix recovery management is installed on the CMS. Error message Unable to validate this 3PAR Storage Replication Group. Cause The parameters entered cannot be validated against those configured on the HP 3PAR storage system. Action Verify and correct the parameters (including volume group name on the local HP 3PAR storage system, volume group name on the remote HP 3PAR storage system, local array remote copy group name, remote array remote copy group name, local and remote HP 3PAR storage system serial numbers, and replication mode) and retry the operation. Configuration error messages 57 Error message Unable to run Matrix recovery management operations because Matrix recovery management Job is in progress or another Matrix recovery management configuration operation is in progress. Cause If an Activate or Deactivate operation is in progress, no configuration operation is allowed, because the Job is in progress. If a Matrix recovery management configuration operation is in progress, no other Matrix recovery management configuration operations are allowed. Action Ensure that the failover process is not taking longer than usual and that the backend job process hp_lsdt_automation.exe is running. Wait for the Activate or Deactivate operation to finish. Error message Unable to import Storage Management Servers. Cause A Storage Management Server is not properly configured in the HP Matrix Operating Environment user interface. Action Ensure that the Storage Management Server is discovered in the HP Matrix Operating Environment user interface. Ensure that Open SSH is installed and configured on the Storage Management Server. If not, run Configure→Configure or Repair Agents from the HP Matrix Operating Environment user interface. Ensure that the credentials for the user are also the login credentials for the server in the HP Matrix Operating Environment user interface. Error message Import Failed. Cause Possible causes include an invalid import file, HP Logical Server Automation service is not running, or one or more recovery logical servers are in an active state. Action Ensure that a valid file exported from the Local Site is used to import the Matrix recovery management configuration at the Remote Site. Confirm that the HP Logical Server Automation service is running on the CMS. Ensure that all recovery logical servers to be managed by Matrix recovery management are in a deactivated state at the time of the Matrix recovery management configuration import. Error message Import succeeded but not all storage managers have been fully configured. Check Matrix recovery management log files for details. Cause Credentials were not configured for one or more remote Storage Management Servers. Action 1. Look at the mxdomainmgr.*.log file to find the remote Storage Management Servers with credentials that were not configured. 2. Make sure that the Storage Management Servers are up and healthy. 3. Discover the Storage Management Servers on the CMS. 4. Click on the Matrix recovery management user interface Storage Management Servers tab. 5. Select the Storage Management Server that you want to configure, and click Edit. 6. Select the Refresh SIM Password box and click Save. Warning messages This section lists Matrix recovery management warning messages. 58 Warning message Warning: Matrix recovery management is being quiesced. Currently running operations will be allowed to complete. No new operations can be started. Cause Matrix recovery management is being quiesced. All configuration buttons (Create, Edit, Delete, etc…) are disabled. Action Wait for Matrix recovery management to be unquiesced. Troubleshooting Warning message Warning: Matrix recovery management is quiesced. No new operations will be allowed. Cause Matrix recovery management has been quiesced. All configuration buttons (Create, Edit, Delete, etc…) are disabled. Action Wait for Matrix recovery management to be unquiesced. Warning message Warning: Unable to remove CLX credentials for (these server credentials may not exist in CLX). Matrix recovery management will continue to delete . HP recommends that you check and manually remove the CLX credentials. Cause Failure to clean a CLX credential after successfully deleting a Storage Management Server. Action To check and clean a CLX credential, open a command prompt window on the CMS: • Change to the CLX bin directory. For example, cd c:\Program Files\Hewlett-Packard\Cluster Extension 3PAR\bin • Execute CLX3PARCONFIG array /remove name= • Check the command output message. Matrix recovery management Job troubleshooting Failure of a Matrix recovery management Activate, Deactivate, Recovery Group import or Site import Job, may intermittently occur due to a transient error condition between the software components. By default, Matrix recovery management automatically retries failed activation and deactivation operations on logical servers. If a Recovery Group import or site import Job fails, there is no automatic retry, and the restart option is not available. The details of each failed Job are logged to the lsdt.log file under the Matrix recovery management installed directory, for example, C:\Program Files\HP\Insight Recovery\logs. If the lsdt.log indicates that a site import Job failed due to one or more Recovery Groups that were not successfully imported, you can attempt to complete the site import by importing the individual Recovery Groups that were not successfully imported in the site import operation. Individual Recovery Groups can be imported from the Recovery Groups tab. If the cause of the site import Job failure is more extensive, you will need to return to the Sites tab and redo the entire site import. If you choose to disable automatic retry, you can edit the MAX_RETRY_ATTEMPTS property in the hp_ir.properties file in the conf directory where Matrix recovery management is installed. To disable automatic retry, set the MAX_RETRY_ATTEMPTS property to 0. To re-enable the automatic retry of failed operations by Matrix recovery management, set the MAX_RETRY_ATTEMPTS property to 1. NOTE: • Failed activate and deactivate Jobs will be retried only after the operation has been attempted on all of the logical servers in the configuration. • Job states for logical server and Recovery Group Jobs will not be marked failed when a retry of the operation is pending. To identify the subsystem where the failure occurred, review the remainder of this section for information on which log files you need to look at and what action you need to take. For additional information, you can view the lsdt.log file in the logs directory where Matrix recovery management is installed on the system. A Job in Matrix recovery management is an automated, multistep process that activates or deactivates selected Recovery Group Sets on a site, imports a specific Recovery Group, or imports a Site configuration. For example, the Job with a Job Id of 3288 shown in Figure 23 (page 60) is Matrix recovery management Job troubleshooting 59 an Activate Job. It has an Entity of type site and an Operation of type activate. You will also notice the Failed icon in the Status column indicating that Job 3288 has failed. Figure 23 Jobs screen For a failed Job, click the check box next to the Job Id to get detailed information about the associated Sub Jobs. A site Job contains a Sub Job for each Recovery Group. Similarly, each Recovery Group has Sub Jobs for its Storage Replication Group and logical server, respectively. To troubleshoot a site Job and identify the source of the error, drill down to each of the associated failed Sub Jobs to determine what operation failed and the reason for the failure, as shown in Figure 24 (page 60). Figure 24 Expanded Jobs screeen For additional information on troubleshooting, see the HP Matrix Operating Environment 7.3 Logical Server Management User Guide and the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. After the problem has been corrected, from the Jobs screen you can restart the Job by clicking restart in the Status column for the failed Job, as illustrated in Figure 25 (page 61). 60 Troubleshooting Figure 25 Restarting a failed job NOTE: Restarting the Job retries only Sub Jobs that previously failed; servers associated with completed Jobs or Sub Jobs are not impacted. IMPORTANT: If correcting the problem that caused the Job to fail included reconfiguration of logical servers, before you restart the Job, go to the Recovery Groups tab and delete the Recovery Groups that contain the reconfigured logical servers. Recovery Groups that have been deleted due to the reconfiguration of logical servers can be recreated with the reconfigured logical servers after the Job is successfully completed. The following troubleshooting issues are addressed in this section: • Failover job failed because storage failover failed. Possible causes include: ◦ • Storage Management Servers were not available at the time of the failover. Failover job succeeded but recovery logical servers are not activated. Possible causes include: ◦ • Recovery Groups that contain logical servers that are in Maintenance Mode at the Remote Site. Failover job failed because there are no sufficient licensed physical servers or virtual machines to host the logical servers. Possible causes include: ◦ Physical servers are running other workloads. ◦ Hypervisor management software (for example, VMware vCenter Server) is not running on the Remote Site. Matrix recovery management Job troubleshooting 61 • Matrix recovery management job failed because of unlocatable logical server in Matrix OE logical server management. Possible causes include: ◦ • A logical server managed by Matrix recovery management was removed from Matrix OE logical server management before it was unmanaged in Matrix recovery management. Matrix recovery management job failed because an operation failed in Matrix OE logical server management for the logical server. Possible causes include: ◦ Power on of the logical server may have failed. Failover error messages This section contains Matrix recovery management failover error messages. Error message Failover job fails because storage failover of Storage Replication Groups failed Cause Possible causes include: The Storage Management Server was not being accessible during the time of failover, or the status of the Storage Replication Groups does not permit a storage failover. Action Ensure that at least the local Storage Management Server is accessible and actively managing one of the arrays. To further triage the problem: • If you are using HP P6000 disk arrays, view the clxevarun.log in STORAGE/EVA/log located where Matrix recovery management is installed on the CMS. • If you are using HP P9000 disk arrays, view clxrun.log in the logs directory where CLX for HP P9000 is installed on the local Storage Management Server. • If you are using HP 3PAR storage systems, view clx3parrun.log in the logs directory where HP 3PAR Cluster Extension Software is installed on the CMS. 62 Error message Failover job fails because of unlocatable logical server in LSM Cause A logical server may have been deleted using Matrix OE visualization without removing the Recovery Group containing the logical server from Matrix recovery management. Action Remove the logical server from the Recovery Group in Matrix recovery management and rerun the failover job. Also look at the lsdt.log file located in the logs directory where Matrix recovery management is installed. Error message Failover job succeeds but certain logical servers are not activated Cause Recovery Groups containing those logical servers may be in Maintenance Mode. Action Disable Maintenance Mode on the Recovery Groups and rerun the failover operation. Also look at the lsdt.log file located in the logs directory where Matrix recovery management is installed. Error message Failover job fails because of insufficient servers Cause There may not be enough licensed physical resources capable of hosting the logical servers. Action Ensure that there are enough physical servers licensed for the HP Matrix Operating Environment available to host the logical servers managed by Matrix recovery management. Also look at the lsdt.log file located in the logs directory where Matrix recovery management is installed. Troubleshooting Matrix recovery management log files There are several log files available with detailed information that you can view to help identify the sources of Matrix recovery management failover or failback problems: • For errors that occur during the initial Matrix recovery management configuration steps, view the mxdomainmgr(0).log file located in the logs directory where HP Systems Insight Manager is installed on the system. • For errors that occur during a failover, check the lsdt.log file in the logs directory where Matrix recovery management is installed for details on the specific operation that failed. • HP P6000 Storage failover related errors are written to the clxevarun.log file in the STORAGE/EVA/log directory located where Matrix recovery management is installed on the system. For a list of log files to use for troubleshooting HP P9000 storage failover issues, see the HP P9000 Cluster Extension Software Administrator Guide. • HP P9000 Storage failover related errors are written to the clxrun.log file located in the Cluster Extension XP\log directory of the storage management server accessed by Matrix recovery management to perform the storage failover operation. Additional information on logical server activation and deactivation issues initially reported in the Matrix recovery management Jobs screen can be found in the Matrix OE visualization Logical Server Job Status screen. Select a logical server job based on its Job Title and view its Job Details. • For a list of log files to use for troubleshooting HP 3PAR storage failover issues, see the HP 3PAR Cluster Extension Software Administrator Guide available at http:// h20000.www2.hp.com. Click Manuals, then go to Storage→Storage Software→Storage Replication Software→HP Cluster Extension Software. DR Protected IO serivces troubleshooting For failures that occur during Matrix recovery management operations on IO services Recovery Groups, the following general troubleshooting steps apply: 1. Check the lsdt.log file in the logs directory where Matrix recovery management is installed for details on the specific operation that failed. 2. Check the status of the IO services in the Matrix IO Services page to see if the IO services are in the correct state for the specific operation that failed. 3. Check the hpio.properties files and the dr.properties files on the Local and Remote Site CMS to make sure they are set correctly. For more information see “DR protection for IO services” (page 23). 4. If an activate, deactivate, or import operation failed, go to the Matrix recovery management Jobs screen and review the Job details. 5. If necessary, go to the IO Requests screen to view the requests on the IO services involved in the failed Matrix recovery management operation. 6. If necessary, check the hpio-controller.log file in the logs directory where IO is installed for details on the specific IO operation that failed. 7. For additional information on troubleshooting issues related to IO services, see the Troubleshooting section in the HP Matrix Operating Environment 7.3 Infrastructure Orchestration User Guide available at Enterprise Information Library. Matrix recovery management log files 63 DR Protected IO services configuration troubleshooting In addition to the configuration issues addressed in this User Guide that are common to both logical servers and IO services, the following configuration issues apply to IO services only: • Failed to get a list of IO services that can be included in a recovery group Possible Causes: • ◦ Matrix infrastructure orchestration Windows service is not running. ◦ There are no IO services that are DR protection enabled. ◦ All DR protection enabled IO services are already configured in IO services Recovery Groups. ◦ Communication failure with IO. Failed to import an IO service Possible causes: • ◦ The import file is invalid. ◦ The replica service to be created already exists and is running locally. ◦ A primary service with the same name already exists. ◦ The values of the properties in the hpio.properties and dr.properties files are not correctly set. – DR protection is not enabled (enable.dr.protection = false). – The datastores specified in volume.dr.replica.list do not match the datastores for the IO service at the Remote Site. – User name and domain name mapping is not specified. ◦ Resources (network and storage) are not available. ◦ The Matrix infrastructure orchestration Windows service is not running. ◦ There is a communication failure with IO. One or more Recovery Groups in the Matrix recovery management configuration are inconsistent with the Matrix OE logical server management or IO configuration Possible causes include: 64 ◦ The Matrix OE logical server management service, the Matrix infrastructure orchestration Windows service, or both of these services were not running when Recovery Groups were configured in Matrix recovery management. ◦ The logical sever or IO service was being actively managed when the Matrix recovery management configuration was invoked (modification, activation, or deactivation was in progress). Troubleshooting IO services configuration error messages Error message Unable to get the IO service. Cause The IO service does not exist or Matrix recovery management failed to get the IO service information from the Matrix infrastructure. Action Check the Matrix recovery management and IO log files for more information on the failure. If the IO service does not exist in IO, it is possible that the IO service was removed. If the IO service exists, restart IO and retry the operation. Error message Error: HP Matrix infrastructure orchestration is down, causing either an empty list of services, or an unfiltered list of available logical servers (which includes the ones managed by Matrix IO services). Restart HP Matrix infrastructure orchestration and retry. Cause When an IO services Recovery Group was created or modified using the Matrix recovery management GUI, Matrix recovery management was not able to communicate with IO to obtain a list of available IO services to be included in the Recovery Group. Action Restart the Matrix infrastructure orchestration Windows service and retry the operation. Error message Error: Failed to filter out logical servers managed by Matrix IO services. Restart Matrix IO and retry. Cause When an IO services Recovery Group was created or modified using the Matrix recovery management GUI, Matrix recovery management was not able to communicate with IO to obtain a list of IO services to filter out the VM logical servers that are not already part of an IO service that has DR Protection enabled. Action Restart the Matrix infrastructure orchestration Windows service and retry the operation. Error message Error: Failed to import Matrix IO service. A primary service with the same name already exists. Cause During import, a primary IO service with the same name as the imported replica IO service exists in IO. Action Either delete the primary IO service at the Local Site or recreate the primary IO service with a different name. Error message Error: One or more Recovery Groups in the Matrix recovery management configuration were found to be inconsistent with the LSM or Matrix infrastructure orchestration configuration. Please delete them. Check the Matrix recovery management log files to identify the inconsistency that caused the error. Cause Matrix recovery management was not able to communicate with Matrix Operating Environment or IO to confirm that a logical server or IO service is being managed by Matrix recovery management. Action Delete the Recovery Group containing the logical server or IO service from Matrix recovery management. Confirm that the Matrix OE logical server management service and the IO service are running. Ensure that the logical server and IO service are not being actively modified by another user. Retry adding the Recovery Group in Matrix recovery management. DR Protected IO serivces troubleshooting 65 DR Protected IO services failover troubleshooting In addition to the failover issues addressed in this User Guide that are common to both logical servers and IO services, the following failover issues apply to IO services only: • Failed to activate IO service in a Recovery Group Possible Causes: • ◦ Storage resources are not available. ◦ The IO service is in an invalid state for activation. ◦ The IO service does not exist. ◦ The Matrix infrastructure orchestration Windows service is not running. ◦ There is a communication failure with IO. Failed to deactivate IO service in a Recovery Group Possible Causes: ◦ The Matrix infrastructure orchestration Windows service is not running. ◦ The IO service does not exist. ◦ The IO service is in an invalid state for deactivation. ◦ There is a communication failure with IO. Matrix recovery management error messages specific to IO services Error message Failed to get the IO service. Cause Matrix recovery management was not able to get the IO service information from IO. Action Check the Matrix recovery management and IO log files for more details on the failure. Restart the Matrix infrastructure orchestration Windows service and retry the operation. Error message Service activation/deactivation cannot be performed at this time since the service is in IN_PROGRESS state. Cause The IO service was being operated on when the Matrix recovery management activation or deactivation operation was performed (modification, activation, or deactivation was in progress). Action Wait for the IO service to exit the IN_PROGRESS state and retry the activation or deactivation operation. Hyper-V based IO service or logical server fails activation Symptom: Hyper-V based Matrix service or logical server fails activation with the following error in Matrix infrastructure orchestration or Matrix OE visualization jobs screen: Unable to bring resource online. (Virtual Machine Management error code: -1) Possible cause: The Device Manager in the Hyper-V cluster node lists the disk devices previously failed over to the remote site as pseudo devices. This can cause Windows disk management not to see the devices that have been made available as read-write as a result of storage failover. One of the reasons for this symptom may be the way multi-path I/O (MPIO) re-presents the same LUN after a failover and failback. Suggested Actions: The following are two workarounds: • 66 Perform a reboot of the cluster nodes that are failing to see the disks, even after the storage replication group has been failed over to the local site (as part of Matrix RM failover) to fix Troubleshooting the issue. After the reboot is complete, check the cluster disks that are visible in the Microsoft Failover Cluster Manager UI and restart the Matrix RM activate operation. • Uninstall the ghosted pseudo devices that are left behind as a result of storage failover to remote site, followed by a rescan for storage. These pseudo devices can be seen in Device Manager’s disk drives (ensure that the hidden devices are seen only when View option is selected. NOTE: The uninstall step must be done cautiously to ensure that only the ghosted disks are uninstalled and not the other valid disks. Hyper-V based IO service or logical server fails activation 67 7 Support and other resources Information to collect before contacting HP Be sure to have the following information available before you contact HP: • Software product name • Hardware product model number • Operating system type and version • Applicable error message • Third-party hardware or software • Technical support registration number (if applicable) How to contact HP Use the following methods to contact HP technical support: • In the United States, see the Customer Service / Contact HP United States website for contact options: http://welcome.hp.com/country/us/en/contact_us.html • In the United States, call 1-800-334–5144 to contact HP by telephone. This service is available 24 hours a day, 7 days a week. For continuous quality improvement, conversations might be recorded or monitored. • In other locations see the Contact HP Worldwide website for contact options: http:// welcome.hp.com/country/us/en/wwcontact.html Security bulletin and alert policy for non-HP owned software components Open source software (such as OpenSSL) or third-party software (such as Java) are sometimes included in HP products. HP discloses that the non-HP owned software components listed in the Insight Management end user license agreement (EULA) are included with Insight Management. The EULA is included with the Insight Management Installer on Insight Management DVD #1. HP addresses security bulletins for the software components listed in the EULA with the same level of support afforded HP products. HP is committed to reducing security defects and helping you mitigate the risks associated with security defects when they do occur. When a security defect is found, HP has a well defined process that culminates with the publication of a security bulletin. The security bulletin provides you with a high level description of the problem and explains how to mitigate the security defect. Subscription service HP recommends that you register your product at the Subscriber's Choice for Business website: http://www.hp.com/country/us/en/contact_us.html After registering, you will receive email notification of product enhancements, new driver versions, firmware updates, and other product resources. Registering for software technical support and update service HP Insight Management components and software products include one year of 24 x 7 HP Software Technical Support and Update Service. This service provides access to HP technical resources for assistance in resolving software implementation or operations problems. The service also provides access to software updates and reference manuals, either in electronic form or on physical media, as they are made available from HP. Customers who purchase an electronic license are eligible for electronic updates only. 68 Support and other resources With this service, Insight Management customers benefit from expedited problem resolution as well as proactive notification and delivery of software updates. For more information about this service, see the following website: http://www.hp.com/services/insight. Registration for this service takes place following online redemption of the license certificate. There are two methods for registering: • If you received a license entitlement certificate, automated registration for this service takes place upon online redemption of the license certificate/key. • If the license information you received for your product instructs you to register for Software Technical Support and Update Service, you must follow the instructions in order to be eligible for telephone support and product updates. How to use your software technical support and update service As HP releases updates to software, the latest versions of the software and documentation are made available to you. The Software Updates and Licensing portal gives you access to software, documentation, and license updates for products on your HP software support agreement. You can access this portal from the HP Support Center: http://www.hp.com/go/hpsc After creating your profile and linking your support agreements to your profile, see the Software Updates and Licensing portal at http://www.hp.com/go/hpsoftwareupdatesupport to obtain software, documentation, and license updates. HP authorized resellers For the name of the nearest HP authorized reseller, see the following sources: • In the United States, see the HP U.S. service locator website: http://www.hp.com/service_locator • In other locations, see the Contact HP worldwide website: http://welcome.hp.com/country/us/en/wwcontact.html Related information For support, software updates, and additional information on Matrix recovery management and other products used with Matrix recovery management, see the following websites: • Matrix recovery management website at http://www.hp.com/go/insightrecovery. • HP Insight Orchestration website at http://www.hp.com/go/insightorchestration. • HP Matrix Operating Environment website at http://www.hp.com/go/matrixoe. • HP Insight Control website at http://www.hp.com/go/insightcontrol. • HP Insight Control virtual machine management website at http://www.hp.com/go/vmmanage . • HP Insight Control server deployment website at http://www.hp.com/go/rdp. • VMware Documentation at http://www.vmware.com/support/pubs. HP authorized resellers 69 Matrix recovery management documentation For more information on Matrix recovery management, see the following sources: • HP Insight Management 7.3 Update 1 Support Matrix Provides Matrix recovery management support information along with other HP Insight hardware, software, and firmware support information. Available at Enterprise Information Library. • HP Matrix Operating Environment 7.3 Release Notes Provides information on what's new with this release, features, and change notifications for Matrix recovery management and other HP Matrix Operating Environment components. Available at Enterprise Information Library. • HP Matrix Operating Environment 7.3 Getting Started Guide Provides installation information for Matrix recovery management and other HP Matrix Operating Environment components. Available at Enterprise Information Library. • HP Matrix Operating Environment 7.3 Recovery Management User Guide Provides information on Matrix recovery management installation, configuration, testing, and troubleshooting. Available at Enterprise Information Library. • Matrix recovery management white papers Matrix recovery management white papers are available at Enterprise Information Library. • Matrix recovery management online help The Matrix recovery management online help provides information on operations that are performed from the Matrix recovery management user interface. It is accessible from the Matrix recovery management user interface and from the Help menu on the HP Matrix Operating Environment home page. Typographic conventions This document uses the following typographical conventions: 70 Book title The title of a book. On the web, this can be a hyperlink to the book itself. Command A command name or command phrase, for example ls -a. Computer output Information displayed by the computer. Ctrl+x or Ctrl-x A key sequence that indicates you must hold down the keyboard key labeled Ctrl while you press the letter x. ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. Key The name of a keyboard key. Return and Enter both refer to the same key. Term A term or phrase that is defined in the body text of the document, not in a glossary. User input Indicates commands and text that you type exactly as shown. Replaceable The name of a placeholder that you replace with an actual value. [] In command syntax statements, these characters enclose optional content. {} In command syntax statements, these characters enclose required content. | The character that separates items in a linear list of choices. ... Indicates that the preceding element can be repeated one or more times. Support and other resources WARNING An alert that calls attention to important information that, if not understood or followed, results in personal injury. CAUTION An alert that calls attention to important information that, if not understood or followed, results in data loss, data corruption, or damage to hardware or software. IMPORTANT An alert that calls attention to essential information. NOTE An alert that contains additional or supplementary information. TIP An alert that provides helpful information. Related information 71 A Recover the CSV from online pending state If you perform an activate operation at the remote site without taking the CSV offline at the local site, you might see the following symptoms at the local site: • No storage information is available when navigating to the storage view in the Windows Failover Cluster Management tool. • The CSV is in the online pending state. To recover from these symptoms, perform the following steps: 1. Using the storage management tool of your choice, perform the following steps: 1. Unexport the CSV virtual volume from the Hyper V cluster. Note the LUN number. Leave the virtual volume unexported until an opportune maintenance period. 2. During the maintenance period, deactivate the recovery group on the other site where it might be currently active and take the CSV offline. 3. Perform storage failover to place the CSV virtual volume in read-write mode. 4. Export the CSV virtual volume to the Hyper V cluster using the same LUN number as when you originaqlly exported the LUN. 2. Bring the CSV online using the Hyper V cluster manager. 3. Rename the CSV folder name to the original folder name. 4. Perform Logical Server refresh resources on the Matrix OE visualization GUI. 5. Activate the recovery group on the CMS. 72 Recover the CSV from online pending state B Documentation feeback Documentation feeback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any error, suggestions, or comments to Documentation Feedback ([email protected]). Include the document title and part number, version number, or the URL when submitting your feedback. Documentation feeback 73 Glossary bidirectional failover A Matrix recovery management feature that allows Recovery Group Sets to be activated or deactivated at either the Local Site or the Remote Site. At any point in time there can be activated and deactivated Recovery Group Sets at both sites. In the event of a disaster, or to accommodate site maintenance, all of the Recovery Group Sets in the Matrix recovery management configuration can be deactivated at one site, and activated at the other site. CMS HP Systems Insight Manager (HP SIM) Central Management Server—A system in the management domain that executes the HP SIM software. All central operations within HP SIM are initiated from this system. consistency group Consistency groups are an important property of asynchronous mode volumes. A consistency group is a group of LUNs that need to be treated the same from the perspective of data consistency (I/O ordering). A consistency group is equal to a device group in the Raid Manager configuration file. cross-technology logical servers Logical servers that can be moved back and forth between being VC-hosted (hosted by a physical server) or VM-hosted (hosted by a virtual machine). See VC-hosted logical server and VM-hosted logical server for more information. CSV Cluster shared volumes. discovery A feature within a management application that finds and identifies network objects. In HP management applications, discovery finds and identifies all the HP systems within a specified network range. DR Group (Disaster Recovery Group) The HP P6000 Continuous Access Software term for a Storage Replication Group. DR Protected Logical servers and Matrix infrastructure orchestration services that are managed by Matrix recovery management are referred to as DR Protected (disaster recovery protected) logical servers and IO services. HP Matrix OE logical server management A component of the HP Matrix Operating Environment software that manages and automates operations associated with logical servers, including resource provisioning, startup, and shutdown. HP SIM HP Systems Insight Manager IO services HP Matrix Operating Environment infrastructure orchestration services. IO services logical server A logical server that is associated with IO services. Job An automated, multistep process associated with: • A Matrix recovery management Activate or Deactivate operation. • A Matrix recovery management Site import operation. • A Matrix recovery management Recovery Group import operation. Local Site In a Matrix recovery management configuration, this is the set of managed nodes and corresponding CMS that your browser is interacting with. logical server A logical server is a management abstraction that simplifies and optimizes the provisioning, management, and movement of a server, whether it is a physical server or a virtual machine. A logical server is the complete software image of a server, including operating system, applications, configuration data, and metadata that you create and assign to operate on a physical server or virtual machine. Logical servers are managed by HP Matrix OE visualization. Maintenance Mode Maintenance Mode is used to test Recovery Groups to ensure they will function properly when an Activate operation is performed. Maintenance mode temporarily stops Matrix recovery management from managing a DR Protected logical server or IO service. One use of maintenance mode is to perform failover rehearsal. By setting Maintenance Mode on a Recovery Group, all of the logical servers and Matrix infrastructure orchestration services in that Recovery Group can be activated through the Matrix Operating Environment. Once you are satisfied with the failover 74 Glossary rehearsal, the Recovery Group and its corresponding logical servers and IO services can be brought back under the control of Matrix recovery management. Matrix infrastructure orchestration services Matrix infrastructure orchestration services (IO services) quickly provision infrastructure to automatically activate physical and virtual servers, storage, and networking from pools of shared resources. More information on Matrix infrastructure orchestration is available at http:// www.hp.com/go/insightorchestration. Matrix recovery management configuration process The following table lists the six steps in the Matrix recovery management configuration process. Table 1 The Matrix recovery management configuration process Step number Description 1 Local and Remote Sites are defined in this step, including naming the sites, designating Central Management Servers, and designating preferred target types (physical servers or virtual machines). 2 Local and remote Storage Management Servers are configured in this step. These servers manage the Storage Replication groups at the Local and Remote Site respectively. 3 Storage Replication Group information is configured in this step. In Matrix recovery management, a Storage Replication Group is a generic term for what is referred to as a DR Group in HP P6000 Continuous Access terminology, or a Device Group in HP P9000 terminology. 4 Recovery Groups are configured in this step. A Recovery Group is a Matrix recovery management concept that pairs one or more logical servers or IO services with a single Storage Replication Group. Recovery Group Sets can be failed over between Local and Remote Sites. A Recovery Group Set includes all Recovery Groups that share the same Preferred and Secondary Sites. 5 Export the Matrix recovery management configuration to a file at the Local Site . 6 Import the Matrix recovery management configuration file at a Remote Site. NPIV N_Port ID Virtualization planned failover A failover of all Recovery Group Sets from one site to another site, initiated in anticipation of an imminent disaster or for planned downtime. portability group A portability group is a group of virtual machines or c-Class blades equipped with HP Virtual Connect (or a combination virtual machines and c-Class blades) that a logical server can be moved between. Power-Up Delay A Recovery Group configuration setting that determines the minimum time delay (in minutes) between the power-up of any two logical servers in a Recovery Group. The actual time delay may be more than the minimum time delay specified. Preferred Site The Preferred Site is the site where you prefer the Recovery Group to be activated unless circumstances require activation of the Recovery Group at the Secondary Site. primary IO service A primary IO service is the first instance of an IO service that you select for DR Protection in a Matrix recovery management configuration. A replica IO service is a service Matrix recovery management creates during an import operation at a recovery site based on the primary IO service definition in the Matrix recovery management export file. private A subnet that is not routed outside the data center and typically contains addresses only in the 192.x.x.x or 10.x.x.x address ranges. A subnet that is accessible to the Internet and cannot contain IP addresses in the 192.x.x.x or 10.x.x.x address ranges. RDM Raw Disk Mapping recoverable IO services Matrix infrastructure orchestration services that have been configured so they can be included in a DR Protected Matrix recovery management IO services Recovery Group. Recovery Group A pairing of one or more logical servers and a single Storage Replication Group. A Recovery Group has a Recovery Group Start Order number associated with it. During a site failover, Recovery Groups are failed over from one site to another site in the order specified. 75 Recovery Group Set A set of Recovery Groups that share the same Preferred and Secondary sites. Recovery Groups cannot be activated or deactivated individually. Instead, all Recovery Groups that share the same Preferred and Secondary site must be activated or deactivated as a set. Recovery Group Sets can be selected for activation or deactivation at the Local site. Recovery Group Start Order An optional number that specifies the order in which a Recovery Group is to be started during a site failover. Recovery Groups without a start order number are started after all Recovery Groups that have an associated start order number. recovery logical servers Logical servers at the Remote Site that are included in a Recovery Group. They are associated with logical servers at the Local Site that are included in the same Recovery Group. Normally they are deactivated and disabled. They are enabled and activated upon site failover. redundant SAN The duplication of components to prevent failure of the SAN solution. Remote Site A site in a Matrix recovery management configuration that is not the Local Site. replica IO service A replica IO service is a service Matrix recovery management creates during an import operation at a recovery site based on the exported IO service definition in the Matrix recovery management export file. replica IO service logical servers Logical servers that are associated with replica IO services. SAN A Storage Area Network (or subnetwork) that connects data storage devices with associated data servers. A SAN is typically part of an overall network of computing resources. Secondary Site The Secondary Site is the site where you prefer the Recovery Group to be on “standby” (in a deactivated state), unless circumstances require activation of the Recovery Group at the Secondary Site. split-brain Split brain occurs when two or more instances of the same application are active simultaneously, which might lead to data corruption. Storage Decoupling A Matrix recovery management feature that allows failover of the logical servers or IO services in a Recovery Group without failing over the Storage Replication Groups that are associated with those logical servers or IO services. Storage Management Server As part of the Matrix recovery management configuration process, servers that manage HP P6000, HP P9000, HP 3PAR, or User Defined storage devices must be defined. These servers are referred to as Storage Management Servers. Storage Replication Group A set of LUNs across which storage replication preserves write-order at the replication target storage array. In HP P6000 Continuous Access Software terminology, this is known as a DR Group. In HP P9000 Continuous Access Software terminology this is known as a consistency group. Sub Job A Sub Job is component of a Matrix recovery management activate, deactivate, Recovery Group import or Site import Job. For example, a Site activate Job would include a Recovery Group activate Sub Job for each Recovery Group that is activated on that site, and each Recovery Group activate Sub Job would include a logical server activate Sub Job for each logical server in that Recovery Group. unplanned failover A failover of all Recovery Group Sets from one site to another site, initiated in response to an unforeseen event that caused an outage at the site where the Recovery Group Sets where activated. User Defined storage Matrix recovery management provides a User Defined storage adapter interface specification to enable one-step Matrix recovery management failover capability for storage types that are supported by Matrix OE, but not yet integrated with Matrix recovery management. User Defined Storage Management Servers When you install a storage adaptor in your Matrix recovery management configuration for a storage type other than HP 3PAR, HP P6000, or HP P9000, you can define (and edit) Storage Management Servers based on that storage adaptor. These are referred to as User Defined Storage Management Servers in a Matrix recovery management configuration. For example, if you create and install a storage adaptor named EMC, the Storage server type drop-down menu for configuring a Storage Management Server will allow you to select EMC as the storage server type. For more information, see the HP Matrix Operating Environment 7.3 Getting Started Guide available at Enterprise Information Library. VC-hosted logical server A logical server running on a c-Class blade equipped with HP Virtual Connect. 76 Glossary VM-hosted logical server A logical server running on a virtual machine under the control of a hypervisor. 77