Avaya Mps Disk Mirroring Configuration And

Transcript

Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Release 4.1 NN44100-300 Issue 1 August 2013 BINDING CONTRACT BETWEEN YOU AND AVAYA INC. OR THE APPLICABLE AVAYA AFFILIATE (“AVAYA”). © 2013 Avaya Inc. All Rights Reserved. Copyright Notice While reasonable efforts have been made to ensure that the information in this document is complete and accurate at the time of printing, Avaya assumes no liability for any errors. Avaya reserves the right to make changes and corrections to the information in this document without the obligation to notify any person or organization of such changes. Documentation disclaimer “Documentation” means information published by Avaya in varying mediums which may include product information, operating instructions and performance specifications that Avaya generally makes available to users of its products. Documentation does not include marketing materials. Avaya shall not be responsible for any modifications, additions, or deletions to the original published version of documentation unless such modifications, additions, or deletions were performed by Avaya. End User agrees to indemnify and hold harmless Avaya, Avaya's agents, servants and employees against all claims, lawsuits, demands and judgments arising out of, or in connection with, subsequent modifications, additions or deletions to this documentation, to the extent made by End User. Link disclaimer Avaya is not responsible for the contents or reliability of any linked websites referenced within this site or documentation provided by Avaya. Avaya is not responsible for the accuracy of any information, statement or content provided on these sites and does not necessarily endorse the products, services, or information described or offered within them. Avaya does not guarantee that these links will work all the time and has no control over the availability of the linked pages. Warranty Avaya provides a limited warranty on its hardware and Software (“Product(s)”). Refer to your sales agreement to establish the terms of the limited warranty. In addition, Avaya’s standard warranty language, as well as information regarding support for this Product while under warranty is available to Avaya customers and other parties through the Avaya Support website: http://support.avaya.com. Please note that if you acquired the Product(s) from an authorized Avaya Channel Partner outside of the United States and Canada, the warranty is provided to you by said Avaya Channel Partner and not by Avaya. “Software” means computer programs in object code, provided by Avaya or an Avaya Channel Partner, whether as stand-alone products or preinstalled on hardware products, and any upgrades, updates, bug fixes, or modified versions. Licenses THE SOFTWARE LICENSE TERMS AVAILABLE ON THE AVAYA WEBSITE, HTTP://SUPPORT.AVAYA.COM/LICENSEINFO ARE APPLICABLE TO ANYONE WHO DOWNLOADS, USES AND/OR INSTALLS AVAYA SOFTWARE, PURCHASED FROM AVAYA INC., ANY AVAYA AFFILIATE, OR AN AUTHORIZED AVAYA CHANNEL PARTNER (AS APPLICABLE) UNDER A COMMERCIAL AGREEMENT WITH AVAYA OR AN AUTHORIZED AVAYA CHANNEL PARTNER. UNLESS OTHERWISE AGREED TO BY AVAYA IN WRITING, AVAYA DOES NOT EXTEND THIS LICENSE IF THE SOFTWARE WAS OBTAINED FROM ANYONE OTHER THAN AVAYA, AN AVAYA AFFILIATE OR AN AVAYA AUTHORIZED AVAYA CHANNEL PARTNER; AVAYA RESERVES THE RIGHT TO TAKE LEGAL ACTION AGAINST YOU AND ANYONE ELSE USING OR SELLING THE SOFTWARE WITHOUT A LICENSE. BY INSTALLING, DOWNLOADING OR USING THE SOFTWARE, OR AUTHORIZING OTHERS TO DO SO, YOU, ON BEHALF OF YOURSELF AND THE ENTITY FOR WHOM YOU ARE INSTALLING, DOWNLOADING OR USING THE SOFTWARE (HEREINAFTER REFERRED TO INTERCHANGEABLY AS “YOU” AND “END USER”), AGREE TO THESE TERMS AND CONDITIONS AND CREATE A 2 Except where expressly stated otherwise, no use should be made of materials on this site, the Documentation, Software, or hardware provided by Avaya. All content on this site, the documentation and the Product provided by Avaya including the selection, arrangement and design of the content is owned either by Avaya or its licensors and is protected by copyright and other intellectual property laws including the sui generis rights relating to the protection of databases. You may not modify, copy, reproduce, republish, upload, post, transmit or distribute in any way any content, in whole or in part, including any code and software unless expressly authorized by Avaya. Unauthorized reproduction, transmission, dissemination, storage, and or use without the express written consent of Avaya can be a criminal, as well as a civil offense under the applicable law. Third Party Components “Third Party Components” mean certain software programs or portions thereof included in the Software that may contain software (including open source software) distributed under third party agreements (“Third Party Components”), which contain terms regarding the rights to use certain portions of the Software (“Third Party Terms”). Information regarding distributed Linux OS source code (for those Products that have distributed Linux OS source code) and identifying the copyright holders of the Third Party Components and the Third Party Terms that apply is available in the Documentation or on Avaya’s website at: http:// support.avaya.com/Copyright. You agree to the Third Party Terms for any such Third Party Components. Trademarks The trademarks, logos and service marks (“Marks”) displayed in this site, the Documentation and Product(s) provided by Avaya are the registered or unregistered Marks of Avaya, its affiliates, or other third parties. Users are not permitted to use such Marks without prior written consent from Avaya or such third party which may own the Mark. Nothing contained in this site, the Documentation and Product(s) should be construed as granting, by implication, estoppel, or otherwise, any license or right in and to the Marks without the express written permission of Avaya or the applicable third party. Avaya is a registered trademark of Avaya Inc. All non-Avaya trademarks are the property of their respective owners. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries. Downloading Documentation For the most current versions of Documentation, see the Avaya Support website: http://support.avaya.com. Contact Avaya Support See the Avaya Support website: http://support.avaya.com for product notices and articles, or to report a problem with your Avaya product. For a list of support telephone numbers and contact addresses, go to the Avaya Support website: http://support.avaya.com, scroll to the bottom of the page, and select Contact Avaya Support. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Contents Chapter 1: Introduction to Disk Mirroring......................................................................... 5 Introduction............................................................................................................................................... 5 Metadevices.............................................................................................................................................. 5 Mirroring.................................................................................................................................................... 6 State Database Replicas........................................................................................................................... 6 World Wide Name Syntax and Hard Disk Drive (HDD) - WWN................................................................ 7 Collecting Target Information about Mirrored Boot Disks......................................................................... 8 Recording Disk Partition Information......................................................................................................... 8 Solaris Volume Manager Commands and Utilities.................................................................................... 9 Chapter 2: Disk Partitions on an MPS System................................................................. 11 Issues Concerning Disk Mirroring Configuration....................................................................................... 11 Allocating a New Partition (s3).................................................................................................................. 12 The State Database and its Replicas........................................................................................................ 12 Locate Disk Space.................................................................................................................................... 12 Change Disk Partition Sizes...................................................................................................................... 13 Formatting Disk Partitions......................................................................................................................... 16 Determining the Size of Mirror Partitions.................................................................................................. 16 Option 1 (preferred) — Mirror same size as the partition to be mirrored.................................................. 17 Option 2 — Mirror is larger than the partition to be mirrored.................................................................... 17 Copy new modified partition map from HDD0 to HDD1................................................................... 18 Making New File Systems on Newly Created Partitions........................................................................... 18 Creating the State Database and its Replicas.......................................................................................... 18 Chapter 3: MPS Primary Drive Mirror Configuration....................................................... 21 Configuring the MPS to Mirror its Primary Drive....................................................................................... 21 Chapter 4: Disabling Mirroring.......................................................................................... 29 Configuring the MPS to Unmirror its Primary Drive................................................................................... 29 Chapter 5: Recovering from Disk Failure......................................................................... 33 Determining the Type of Disk Failure........................................................................................................ 33 If you are Able to Log into the System...................................................................................................... 33 Determine if you have a Disk or Mirror Failure.......................................................................................... 33 Recovering from a Failed Mirror................................................................................................................ 36 Recovering from a Failed Mirror Disk (Secondary Drive).......................................................................... 37 Check the State Databases...................................................................................................................... 37 Delete Corrupted State Databases........................................................................................................... 38 Replacing a Failed Mirror.......................................................................................................................... 38 Recovering from a Failed Disk.................................................................................................................. 42 If you are Unable to Log In to the System................................................................................................. 46 Recovering from Boot Device Failure....................................................................................................... 47 Recovering from Dual Disk Failure........................................................................................................... 52 If your System Fails to Boot due to a Corrupt /etc/vfstab.......................................................................... 53 mirrorbu Script.................................................................................................................... 57 Overview................................................................................................................................................... 57 Using the mirrorbu Script.......................................................................................................................... 57 Preparing to Use mirrorbu......................................................................................................................... 58 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 3 Running mirrorbu...................................................................................................................................... Recovering from Disk Failure.................................................................................................................... Primary Boot Disk Failure......................................................................................................................... Secondary Boot Disk (Mirror) Failure........................................................................................................ Recovering when All Drives in the System Disk Mirror Fail...................................................................... Restoring Data.......................................................................................................................................... Additional Information............................................................................................................................... mdsk Package Processing........................................................................................................................ Partitioning Guidelines.............................................................................................................................. 4 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD 61 65 65 70 70 74 75 75 76 August 2013 Chapter 1: Introduction to Disk Mirroring This chapter covers: 1. Introduction to Disk Mirroring 2. Collecting Target Information about Mirrored Boot Disks 3. Recording Disk Partition Information Introduction The Solaris Volume Manager, bundled into the Solaris 10 operating system, includes all functionality from the Solstice DiskSuite. The Solaris Volume Manager offers a pseudo device driver (called a metadisk driver) providing better performance, greater capacity, and improved availability of data. The Solaris Volume Manager allows for up to three-way mirroring of most file systems, including /usr, root, /doc,and swap. Other features include online concatenation of physical drives, online expansion of file systems, disk striping, and hot spares. The Avaya Media Processing Server is optionally configured to mirror to one other disk. Metadevices Metadevices are the functional unit of the metadisk driver. After creating metadevices, you can use them like disk partitions. These logical devices can be made up of one or more component partitions. Metadevices are transparent to application software and to component and controller hardware. The standard metadevice name begins with d and is followed by a number. By default, there are 128 unique metadevices in the range 0 to 127. Additional metadevices may be added. Metadevice names are located in /dev/md/dsk and /dev/md/rdsk. When using Solaris Volume Manager commands such as metastat and metainit or when setting up metadevices in the md.tab file, the metadevice name does not need to be fully Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 5 Introduction to Disk Mirroring qualified. For example, you can enter d1 rather than /dev/md/dsk/d1. The examples presented in this manual use the short and long forms for metadevice names interchangeably. Mirroring Solaris Volume Manager supports mirroring to as many as three separate metadevices. This allows the system to tolerate single-component failures with two-way mirroring and double failures with three-way mirroring. Mirroring occurs when a metamirror is created. Metamirrors are a special type of metadevice made up of one or more other metadevices. Each metadevice within a metamirror is called a submirror. Metamirrors are defined in the /etc/lvm/md.tab file, where they are given names such as /dev/ md/dsk/d0. After a metamirror is defined, it becomes active when the metainit utility is invoked. After a metamirror is defined, additional submirrors can be added at a later date without bringing the system down or disrupting writes and reads to existing metamirrors. Submirrors are added by invoking metattach command, which attaches a submirror to the metamirror. When the submirror is attached, all the data from any other submirror in the metamirror is automatically written to the newly attached submirror. This is called resyncing. When the resyncing is complete, the new submirror is readable and writable. After a metattach is performed, the submirrors remain attached (even when the system is rebooted) until they are explicitly detached with the metadetach command. If a component fails, the metadetach command is used to detach the component until a new one is connected. State Database Replicas State database replicas provide the non-volatile memory necessary to keep track of configuration and status information for mirrors and submirrors. The replicas also keep track of error conditions that have occurred. After a metadevice is configured, the metadevice driver must remember its configuration and status information. The metadevice state database is in the metadevice driver's long term memory. The metadevice driver stores all the metadevice configuration information in the state database. This includes the configuration information about mirrors and submirrors. If the replicated metadevice state database is lost, the metadevice driver cannot retrieve any configuration information. This could result in the loss of all data stored on the metadevices. To protect against losing the metadevice state database because of hardware failures, multiple replicas (copies) of the state database are kept. 6 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 World Wide Name Syntax and Hard Disk Drive (HDD) - WWN These multiple replicas also protect the state database against corruption that can result from a system crash. Each replica of the state database contains a checksum. When the state database is updated, each replica is modified one at a time. If a crash occurs while the database is being updated, only one of the replicas will be corrupted. When the system reboots, the metadevice driver uses the checksum embedded in the replicas to determine if a replica has been corrupted. Any replicas that have been corrupted are ignored. If a disk that contains the metadevice state database is turned off, the metadevices remain fully functional because the database is retrieved from one of the replicas still in operation. Changes made to the configuration following the reboot are stored only in the replicas that are in operation when the system is rebooted. If the disk drive that was turned off is later turned back on, the data contained in the replica stored on that disk is ignored. Proper configuration is accomplished through comparison with other replicas. World Wide Name Syntax and Hard Disk Drive (HDD) - WWN Before the change to WWN nomenclature, the Oracle Solaris OS would ordinarily identify the default boot device as cxtxd0. With the change, the device identifier for the default boot device is now referred to as c0tWWNd0, where WWN is a hexadecimal value that is unique to this device throughout the world. This WWN value is assigned by the manufacturer of the device and therefore, has a random relationship to the server’s device tree structure. Example: World Wide Name or World Wide Number Device name: c0tWWNd0 Server Type: Netra SPARC T4-1 Hard Disk Drive: c0t5000CCA0221AD058d0 The Unique WWN is: 5000CCA0221AD058 = 16 characters WWN = 5000CCA0221AD058 Disk Information: # diskinfo -a Enclosure path: Chassis Serial Number: Chassis Model: Label ---------/SYS/HDD0 /SYS/HDD1 1308BD0043-physical-hba-1 1308BD0043-physical-hba-1 ORCL,Netra-SPARC-T4-1 Disk name ---------------------c0t5000CCA0221AD058d0 c0t5000CCA0221AD4ECd0 Vendor -------HITACHI HITACHI Product ---------------H109030SESUN300G H109030SESUN300G Vers ---A31A A31A # Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 7 Introduction to Disk Mirroring Collecting Target Information about Mirrored Boot Disks Several procedures in this guide require specific information concerning disk drives. Use the following procedure to retrieve this information: 1. Verify which target is the alternate boot by viewing md.tab which resides in /etc/lvm on Solaris 10. 2. Search for the line beginning with /dev/md/dsk/d20 1 1 in the md.tab file. In the following sample line, /dev/dsk/c0tWWNd0s0 is the root mirror for the system disk. /dev/md/dsk/d20 1 1 /dev/dsk/c0tWWNd0s0 3. Record the alternate boot path. The alternate boot path is the device link of the boot mirror in the output of the following command: # ls -al /dev/dsk/c0tWWNd0s0 Compare your output to the following sample: bash-3.2# ls -la /dev/dsk/c0t5000CCA0221AD058d0s0 lrwxrwxrwx 1 root root 48 Mar 6 19:43 /dev/dsk/ c0t5000CCA0221AD058d0s0 -> ../../devices/scsi_vhci/ disk@g5000cca0221ad058:a bash-3.2# Recording Disk Partition Information Caution: In case of disk failure, you may have to reboot from a mirrored disk. You may be locked out of your system and unable to get the necessary information at that time. Please record the necessary disk information in the following table for future reference. Typically, Avaya Media Processing Server Series mirror and data disks are on the same controller and the mirror disk boot target can be specified as disk[x] where [x] is the target. For example, disk1 for the disk at target 1. This the case with the NETRA SPARC T4-1 server. To verify your alternate boot device, see Configuring the MPS to Mirror its Primary Drive on page 21, step 7 on page 24, to verify your alternate root device. Record this information when you have configured mirroring, to have it available, if a failure occurs. Disk 8 Disk Controller/ Target/Device/Slice Root Mirror Output Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Solaris Volume Manager Commands and Utilities Root Root Mirror 1 Root Mirror 2 Solaris Volume Manager Commands and Utilities For more information on the Solaris Volume Manager, refer to the MPS Command Reference Manual and the online documents for Solaris 10 on the Sun web site. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 9 Introduction to Disk Mirroring 10 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Chapter 2: Disk Partitions on an MPS System This chapter covers: 1. Issues Concerning Disk Mirroring Configuration 2. Allocating a New Disk Partition 3. Formatting a New Disk Partition 4. Making New File Systems on Newly Created Partitions 5. Creating the State Database and its Replicas Issues Concerning Disk Mirroring Configuration If your system was manufactured by Avaya, it may optionally have mirroring already configured. Execute the metastat command to confirm whether the system is presently mirrored. The following sample shows a system that is not mirrored: # metastat metastat: T4-24-190: there are no existing databases If the output indicates that there are no existing databases, then mirroring has not been configured. Proceed with the steps in this chapter and MPS Primary Drive Mirror Configuration on page 21 to configure the system. Partitions which can not presently be mirrored Caution: Any file system containing multimedia files activated by VMM must have the "diskMirrorSupport on" configuration parameter added to the $MPSHOME/mps<#>/etc/ vmm.cfg file (see the MPS Series Command Reference Manual (NN44100-109)). The customer should still back up all file systems, as specified in the MPS System Operators Guide. Mirroring prevents downtime if a partition fails; it is not considered a substitute form of backup. Number and configuration of replicas State databases (replicas) contain configuration and status information and are required for the Solaris Volume Manager operation. At least three replicas must be created because the majority consensus algorithm is used, and a consensus can be reached if two of the three Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 11 Disk Partitions on an MPS System databases are up. Create three replicas on one disk, and three on another disk so that the Solaris Volume Manager can still function if one disk fails. If the majority (half plus one) of the total number of replicas are not available, you can not boot into multiuser. If you are in a situation where the system will not boot due to quorum not reached, you must boot into single user mode (boot -s) and delete the corrupt databases. For increased reliablilty, additional replicas can be spread across multiple disks. Allocate a minimum of 100 MByte partition (Avaya T4 now defaults at 5 GByte) on an unused slice for the state database and two replicas. Mirroring onto different disk types Different disk types have varying numbers of tracks and sectors per cylinder. The starting and ending cylinder for a certain size partition on one disk may be different than the starting and ending cylinder, for the same size partition, on another disk. See Change Disk Partition Sizes on page 13. Allocating a New Partition (s3) The State Database and its Replicas Important: Avaya Media Processing Server Series systems, manufactured by Avaya, are partitioned to allow for the creation of the state database and its replicas. If your system was not manufactured by Avaya, then you must locate a slice to be dedicated to state database replicas. If necessary, you can remove space from the swap partition and allocate it to an unused slice. If your system does not already have a dedicated slice, use the format command to remove a minimum of 100 MByte but Avaya recommends 1 to 5 GByte if available from the swap partition, and allocate a new partition (s3) for the state database and its replicas. Use the format command to partition your mirror disk. SeeChange Disk Partition Sizes on page 13 Locate Disk Space The procedure to locate disk space is as follows: 1. Shut down the system using the following commands. 12 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Change Disk Partition Sizes #su #halt 2. Bring the system to single user mode. OK> boot -s 3. Determine your swap partition. #swap -l In most cases, the swapfile listed is /dev/dsk/c0tWWNd0s1 4. Turn off swapping to the partition from which you intend to use blocks. # swap -d /dev/dsk/c0tWWNd0s1 # swap -l No swap devices configured # Change Disk Partition Sizes Reformat a partition to be smaller by 5 GByte and use the resulting disk space to create the 5 GByte partition necessary for the state database. Use the format command as shown in the following example. # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t5000CCA0221ACEFCd0 < HITACHI-H109030SESUN300G-A31A cyl 46873 alt 2 hd 20 sec 625> solaris /scsi_vhci/disk@g5000cca0221acefc 1. c0t5000CCA0221ACFF8d0 < HITACHI-H109030SESUN300G-A31A cyl 46873 alt 2 hd 20 sec 625> solaris /scsi_vhci/disk@g5000cca0221acff8 Specify disk (enter its number): 0 selecting c0t5000CCA0221ACEFCd0: solaris [disk formatted] Warning: Current Disk has mounted partitions. /dev/dsk/c0t5000CCA0221ACEFCd0s0 is currently mounted on /. Please see umount(1M). /dev/dsk/c0t5000CCA0221ACEFCd0s1 is normally mounted on according to /etc/vfstab. Please remove this entry to use this device. /dev/dsk/c0t5000CCA0221ACEFCd0s4 is currently mounted on /var. Please see umount(1M). /dev/dsk/c0t5000CCA0221ACEFCd0s5 is normally mounted on /opt according to /etc/ vfstab. Please remove this entry to use this device. /dev/dsk/c0t5000CCA0221ACEFCd0s6 is currently mounted on /usr. Please see umount(1M). /dev/dsk/c0t5000CCA0221ACEFCd0s7 is normally mounted on /mmf according to /etc/ vfstab. Please remove this entry to use this device. FORMAT MENU: disk type partition current format repair - select a disk select (define) a disk type select (define) a partition table describe the current disk format and analyze the disk repair a defective sector Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 13 Disk Partitions on an MPS System label analyze defect backup verify save inquiry volname !< cmd> quit format> part - write label to the disk surface analysis defect list management search for backup labels read and display labels save new disk/partition definitions show vendor, product and revision set 8-character volume name - execute < cmd>, then return PARTITION MENU: 0 - change `0' partition 1 - change `1' partition 2 - change `2' partition 3 - change `3' partition 4 - change `4' partition 5 - change `5' partition 6 - change `6' partition 7 - change `7' partition select - select a predefined table modify - modify a predefined partition table name - name the current table print - display the current table label - write partition map and label to the disk !< cmd> - execute < cmd>, then return quit partition> partition> print Volume: solaris Current partition table (unnamed): Total disk cylinders available: 46873 + 2 (reserved cylinders) Part Tag 0 root 1 swap 2 backup 3 unassigned 4 var 5 unassigned 6 usr 7 unassigned Flag wm wu wm wm wm wm wm wm Cylinders 0 865 866 - 7629 0 - 46872 0 7617 - 14498 14499 - 26459 26460 - 31620 31621 - 46858 Size 5.16GB 40.32GB 279.38GB 0 41.02GB 71.29GB 30.76GB 90.83GB Blocks (866/0/0) 10825000 (6764/0/0) 84550000 (46873/0/0) 585912500 (0/0/0) 0 (6882/0/0) 86025000 (11961/0/0) 149512500 (5161/0/0) 64512500 (15238/0/0) 190475000 partition> 1 Part Tag 1 swap Flag wu Cylinders 866 - 7629 Size 40.32GB Blocks (6764/0/0) 84550000 Enter partition id tag[swap]: Enter partition permission flags[wu]: Enter new starting cyl[866]: Enter partition size[84550000b, 6764c, 7629e, 41284.18mb, 40.32gb]: 35gb partition> print Volume: solaris Current partition table (unnamed): Total disk cylinders available: 46873 + 2 (reserved cylinders) Part Tag 0 root 1 swap 2 backup 3 unassigned 4 var 5 unassigned 14 Flag wm wu wm wm wm wm Cylinders 0 865 866 - 6738 0 - 46872 0 7617 - 14498 14499 - 26459 Size 5.16GB 35.01GB 279.38GB 0 41.02GB 71.29GB Blocks (866/0/0) 10825000 (5873/0/0) 73412500 (46873/0/0) 585912500 (0/0/0) 0 (6882/0/0) 86025000 (11961/0/0) 149512500 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Change Disk Partition Sizes 6 usr 7 unassigned partition> 3 Part Tag 3 unassigned wm wm Flag wm 26460 - 31620 31621 - 46858 Cylinders 0 30.76GB 90.83GB Size 0 (5161/0/0) 64512500 (15238/0/0) 190475000 Blocks (0/0/0) 0 Enter partition id tag[unassigned]: Enter partition permission flags[wm]: Enter new starting cyl[0]: 6739 Enter partition size[0b, 0c, 6739e, 0.00mb, 0.00gb]: 5gb partition> partition> print Volume: solaris Current partition table (unnamed): Total disk cylinders available: 46873 + 2 (reserved cylinders) Part Tag 0 root 1 swap 2 backup 3 unassigned 4 var 5 unassigned 6 usr 7 unassigned Flag wm wu wm wm wm wm wm wm Cylinders 0 865 866 - 6738 0 - 46872 6739 - 7577 7617 - 14498 14499 - 26459 26460 - 31620 31621 - 46858 Size 5.16GB 35.01GB 279.38GB 5.00GB 41.02GB 71.29GB 30.76GB 90.83GB Blocks (866/0/0) 10825000 (5873/0/0) 73412500 (46873/0/0) 585912500 (839/0/0) 10487500 (6882/0/0) 86025000 (11961/0/0) 149512500 (5161/0/0) 64512500 (15238/0/0) 190475000 Flag wm Cylinders 6739 - 7577 Size 5.00GB Blocks (839/0/0) 10487500 partition> partition> 3 Part Tag 3 unassigned Enter partition id tag[unassigned]: Enter partition permission flags[wm]: Enter new starting cyl[6739]: Enter partition size[10487500b, 839c, 7577e, 5120.85mb, 5.00gb]: partition> partition> print Volume: solaris Current partition table (unnamed): Total disk cylinders available: 46873 + 2 (reserved cylinders) Part Tag 0 root 1 swap 2 backup 3 unassigned 4 var 5 unassigned 6 usr 7 unassigned Flag wm wu wm wm wm wm wm wm Cylinders 0 865 866 - 6738 0 - 46872 6739 - 7577 7617 - 14498 14499 - 26459 26460 - 31620 31621 - 46858 Size 5.16GB 35.01GB 279.38GB 5.00GB 41.02GB 71.29GB 30.76GB 90.83GB Blocks (866/0/0) 10825000 (5873/0/0) 73412500 (46873/0/0) 585912500 (839/0/0) 10487500 (6882/0/0) 86025000 (11961/0/0) 149512500 (5161/0/0) 64512500 (15238/0/0) 190475000 partition> q FORMAT MENU: disk type partition current format repair label analyze - select a disk select (define) a disk type select (define) a partition table describe the current disk format and analyze the disk repair a defective sector write label to the disk surface analysis Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 15 Disk Partitions on an MPS System defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show vendor, product and revision volname - set 8-character volume name !< cmd> - execute < cmd>, then return quit format> label Ready to label disk, continue? y format> q # Two state databases (replicas) are required for Solaris Volume Manager to run. A third provides a failsafe if one database becomes corrupt. Therefore, at a minimum, three state databases should be created in one disk and three on another disk. If one disk fails, Solaris Volume Manager will then still function. Spread additional replicas across multiple disks for increased reliability. See Creating the State Database and its Replicas on page 18. Formatting Disk Partitions Format the partition sizes on the second disk identical to or larger than those of the first disk (HDD0). For Netra SPARC T4-1 systems manufactured by Avaya, the system disk is on HDD0 and the mirror disk is on HDD1. Determining the Size of Mirror Partitions The mirror must be at least as large as the partition to be mirrored. If both disks are the same geometry (same disk type), you may specify the size as n MB equal to the size on the first disk. Avaya strongly recommends you have the same geometry on both disks, otherwise there will be a degradation in mirroring performance. If the disks are different types when you are specifying the size in megabytes during a format, the second disk may be rounded to a fractional amount less than the first. To prevent this, use one of the following options to ensure that the mirror is larger than or equal to the data partition: 16 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Option 1 (preferred) — Mirror same size as the partition to be mirrored Option 1 (preferred) — Mirror same size as the partition to be mirrored Calculate the number of cylinders required to hold at least the size of the partition on the first disk. Option 2 — Mirror is larger than the partition to be mirrored If the mirror is larger than the partition to be mirrored because the disks are different sizes, you must do the following: • Calculate the size of the cylinders on each disk. • Allocate one more cylinder to the disk to be mirrored. Warning: Avaya strongly recommends using a mirror the same size as the partition to be mirrored. If not, there will be a degradation in mirroring performance. If your system was manufactured as an MPS single disk, where the multimedia file partition (mmf) is located on the system drive slice 7 instead of a separate disk, make Partition 7 the free hog. The MPS single disk configuration Partition 7 (/mmf) is not mirrored due to its structure and has extra space. Before you execute format, use the prtvtoc command shown below to display the disk geometry and determine how many cylinders you need to allocate for each partition. The following is an example of the prtvtoc command. # * * * * * * * * * * * * * * * * * prtvtoc /dev/dsk/c0t5000CCA0221ACEFCd0s0 /dev/dsk/c0t5000CCA0221ACEFCd0s0 (volume "solaris") partition map Dimensions: 512 bytes/sector 625 sectors/track 20 tracks/cylinder 12500 sectors/cylinder 46875 cylinders 46873 accessible cylinders Flags: 1: unmountable 10: read-only Unallocated space: First Sector Sector Count Last Sector Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 17 Disk Partitions on an MPS System * 94725000 487500 95212499 * 585737500 175000 585912499 * * First Sector Last * Partition Tag Flags Sector Count Sector 0 2 00 0 10825000 10824999 1 3 01 10825000 73412500 84237499 2 5 00 0 585912500 585912499 3 0 00 84237500 10487500 94724999 4 7 00 95212500 86025000 181237499 5 0 00 181237500 149512500 330749999 6 4 00 330750000 64512500 395262499 7 0 00 395262500 190475000 585737499 # Mount Directory / /var /opt /usr /mmf For each partition, type the desired size in megabytes or cylinders according to your determinations in Determining the Size of Mirror Partitions on page 16 Refer to the MPS Command Reference Manual for details about the format command. Copy new modified partition map from HDD0 to HDD1. # prtvtoc /dev/rdsk/c0t5000CCA0221ACEFCd0s2 | fmthard -s - /dev/rdsk/ c0t5000CCA0221ACFF8d0s2 fmthard: New volume table of contents now in place. Making New File Systems on Newly Created Partitions Make new file systems on the new partitions you just created. Use the newfs command. Caution: Do not execute newfs on existing partitions. You will destroy all the data on the existing partitions. # prtvtoc /dev/rdsk/c0t5000CCA0221ACEFCd0s2 | fmthard -s - /dev/rdsk/ c0t5000CCA0221ACFF8d0s2 fmthard: New volume table of contents now in place. Creating the State Database and its Replicas Solaris Volume Manager commands are located in /usr/sbin, found through the default PATH environment variable. Use the following command to create the initial state database and two replicas: # metadb -c 3 -a -f /dev/dsk/c0t5000CCA0221ACEFCd0s3 18 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Creating the State Database and its Replicas When the first state database is created, the following message appears: metadb:There are no existing databases This is expected, as no databases are yet created on the partition. Use the following command to create an additional three replicas on another disk: # metadb -c 3 -a -f /dev/dsk/c0t5000CCA0221ACEFCd0s3 Execute the metadb -i command to check that all six databases are created. # metadb -i flags first blk block count a u 16 8192 /dev/dsk/c0t5000CCA0221ACEFCd0s3 a u 8208 8192 /dev/dsk/c0t5000CCA0221ACEFCd0s3 a u 16400 8192 /dev/dsk/c0t5000CCA0221ACEFCd0s3 a u 16 8192 /dev/dsk/c0t5000CCA0221ACFF8d0s3 a u 8208 8192 /dev/dsk/c0t5000CCA0221ACFF8d0s3 a u 16400 8192 /dev/dsk/c0t5000CCA0221ACFF8d0s3 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors # Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 19 Disk Partitions on an MPS System 20 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Chapter 3: MPS Primary Drive Mirror Configuration This chapter covers: 1. Configuring the Solaris Volume Manager to Mirror the Primary MPS Hard Disk Configuring the MPS to Mirror its Primary Drive How to ... mirror partitions 1. Edit (or create) /etc/lvm/md.tab file. See step 1 on page 22. 2. Invoke the metainit command for each of the metadevices and the metamirrors defined in the /etc/lvm/md.tab file. See step 2 on page 23. 3. Copy the vfstab file to vfstab.save. See step 3 on page 23. 4. Edit the /etc/vfstab file to change certain entries. See step 4 on page 23. 5. Execute the metaroot command. See step 5 on page 23. 6. Reboot the system. See step 6 on page 24. 7. Attach the new metadevice to the one-way mirror. See step 7 on page 24. 8. Determine the command to boot from the alternate root, in case your primary root fails. See step 7 on page 24. 9. Set an alternate boot path to the mirror. See step 8 on page 24. 10. Record the alternate boot device path. 9 on page 24. 11. Define a backup root device alias for the secondary root (/) file system mirror. See 10 on page 24. 12. Check the status of all the mirrors. See 11 on page 25. 13. Update the crash dump facility to use the mirrrored swap meta device. See 11 on page 25. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 21 MPS Primary Drive Mirror Configuration The following example shows the mirroring of slices 0, 1, 4, 5, and 6. These slices correspond to the root, swap, /usr, /opt and /var partitions, on a MPS system manufactured at Avaya. 1. Edit (or create) the /etc/lvm/md.tab file using any editor and include the following: a. metadevices for the existing file systems b. one-way mirrors c. metadevices for the new components on the other halves of the mirrors In the following example, the metadevice names are assigned as follows: /dev/md/dsk/d1n /dev/md/dsk/dn -m /dev/md/dsk/d2n n disk containing data metadevice name of the disk containing data disk containing the mirror of the data. is the slice. # vi md.tab # # Create a metadisk d5 for /opt. # /dev/md/dsk/d5 -m /dev/md/dsk/d15 #/dev/md/dsk/d15 1 1 /dev/dsk/c0tWWNHDD0d0s5 #/dev/md/dsk/d25 1 1 /dev/dsk/c0tWWNHDD1d0s5 /dev/md/dsk/d15 1 1 /dev/dsk/c0t5000CCA0221ACEFCd0s5 /dev/md/dsk/d25 1 1 /dev/dsk/c0t5000CCA0221ACFF8d0s5 # # Create a metadisk d6 for /usr # /dev/md/dsk/d6 -m /dev/md/dsk/d16 #/dev/md/dsk/d16 1 1 /dev/dsk/c0tWWNHDD0d0s6 #/dev/md/dsk/d26 1 1 /dev/dsk/c0tWWNHDD1d0s6 /dev/md/dsk/d16 1 1 /dev/dsk/c0t5000CCA0221ACEFCd0s6 /dev/md/dsk/d26 1 1 /dev/dsk/c0t5000CCA0221ACFF8d0s6 # # Create a metadisk d1 for /swap # /dev/md/dsk/d1 -m /dev/md/dsk/d11 #/dev/md/dsk/d11 1 1 /dev/dsk/c0tWWNHDD0d0s1 #/dev/md/dsk/d21 1 1 /dev/dsk/c0tWWNHDD1d0s1 /dev/md/dsk/d11 1 1 /dev/dsk/c0t5000CCA0221ACEFCd0s1 /dev/md/dsk/d21 1 1 /dev/dsk/c0t5000CCA0221ACFF8d0s1 # # Create a metadisk d4 for /doc # /dev/md/dsk/d4 -m /dev/md/dsk/d14 #/dev/md/dsk/d14 1 1 /dev/dsk/c0tWWNHDD0d0s1 #/dev/md/dsk/d24 1 1 /dev/dsk/c0tWWNHDD1d0s1 /dev/md/dsk/d14 1 1 /dev/dsk/c0t5000CCA0221ACEFCd0s4 22 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Configuring the MPS to Mirror its Primary Drive /dev/md/dsk/d24 1 1 /dev/dsk/c0t5000CCA0221ACFF8d0s4 # # Create a metadisk d0 for / # /dev/md/dsk/d0 -m /dev/md/dsk/d10 #/dev/md/dsk/d10 1 1 /dev/dsk/c0tWWNHDD0d0s1 #/dev/md/dsk/d20 1 1 /dev/dsk/c0tWWNHDD1d0s1 /dev/md/dsk/d10 1 1 /dev/dsk/c0t5000CCA0221ACEFCd0s0 /dev/md/dsk/d20 1 1 /dev/dsk/c0t5000CCA0221ACFF8d0s0 2. Invoke the metainit command for each of the metadevices and the metamirrors defined in the /etc/lvm/md.tab file. Note that a -f option is used with metainit for active components. 3. Copy /etc/vfstab to /etc/vfstab.save. #cp /etc/vfstab /etc/vfstab.save 4. Edit the /etc/vfstab to change the file system entries that were just mirrored: /dev/dsk/c0t5000CCA0221ACEFCd0s1 no /dev/dsk/c0t5000CCA0221ACEFCd0s4 c0t5000CCA0221ACEFCd0s4 /var /dev/dsk/c0t5000CCA0221ACEFCd0s5 c0t5000CCA0221ACEFCd0s5 /opt /dev/dsk/c0t5000CCA0221ACEFCd0s6 c0t5000CCA0221ACEFCd0s6 /usr - - /dev/rdsk/ ufs 1 /dev/rdsk/ ufs 2 /dev/rdsk/ ufs 1 swap - no - yes - no - TO READ: /dev/md/dsk/d1 /dev/md/dsk/d4 /dev/md/dsk/d5 /dev/md/dsk/d6 /dev/md/rdsk/d4 /dev/md/rdsk/d5 /dev/md/rdsk/d6 swap /var /opt /usr ufs ufs ufs no 1 2 1 no yes no - 5. Execute the metaroot command to change the root entry. #metaroot /dev/md/dsk/d0 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 23 MPS Primary Drive Mirror Configuration You must execute metaroot instead of manually updating slice 0 (root) in /etc/vfstab, because root conversion requires additional special processing supplied by the metaroot. Since the root file system has changed, you must reboot (the next step) before making any other changes. 6. Reboot the system. #/usr/sbin/reboot 7. Attach the new metadevice to the one-way mirror using metattach. After the above steps are performed, a re sync of the mirror begins automatically, which copies all the data that exists on d1a to the mirror d2a. Similarly, all metadevices are copied. 8. Boot options. When the mirror disk is on the same controller as the system disk, you can specify disk[n] on the boot command, where [n] is the target of the mirror disk, to boot off the mirror during recovery. 9. SPARC T4-1 boot device path. HDD0 and HDD1 are on the same controller. For example: # diskinfo -a Enclosure path: Chassis Serial Number: Chassis Model: Label ---------/SYS/HDD0 /SYS/HDD1 1308BD0044-physical-hba-1 1308BD0044-physical-hba-1 ORCL,Netra-SPARC-T4-1 Disk name ---------------------c0t5000CCA0221ACEFCd0 c0t5000CCA0221ACFF8d0 Vendor -------HITACHI HITACHI Product ---------------H109030SESUN300G H109030SESUN300G Vers ---A31A A31A # ls -l /dev/dsk/c0t5000CCA0221ACEFCd0s0 lrwxrwxrwx 1 root root 48 Mar 8 14:40 /dev/dsk/ c0t5000CCA0221ACEFCd0s0 -> ../../devices/scsi_vhci/ disk@g5000cca0221acefc:a # ls -l /dev/dsk/c0t5000CCA0221ACFF8d0s0 lrwxrwxrwx 1 root root 48 Mar 8 14:40 /dev/dsk/ c0t5000CCA0221ACFF8d0s0 -> ../../devices/scsi_vhci/ disk@g5000cca0221acff8:a # 10. Define a backup root device alias for the secondary root (/) file system mirror. Use the OpenBoot™ PROM nvalias command to redefine the boot-device alias to reference both the primary and secondary submirrors. Execute the commands in 24 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Configuring the MPS to Mirror its Primary Drive the order in which you want the submirrors to be used. Substitute the alternate boot device path you have just determined in the command. Note: use the SASAddress of the HDD. Not the SASDeviceName. Probe-scsi-all will make HDD information clear to the user For example: (SASDeviceName 5000cca0221acff8 SASAddress 5000cca0221acff9 ok nvalias backup-root /pci@400/pci@2/pci@0/pci@4/scsi@0/ disk@w5000cca0221acff9,0:a PhyNum 1) Store the configuration. ok printenv boot-device boot-device = disk net ok setenv boot-device disk backup-root net boot-device = disk backup-root net ok nvstore If the primary submirror fails, now the system automatically boots to the second submirror. To boot manually, rather than using autoboot, enter the following command: ok boot backup-root 11. Check the status of all the mirrors. #metastat If you have just issued the metainit command, the state may temporarily show as: Resync After some time, the report shows the status of all the metamirrors and submirrors as Okay. Depending upon the size of the disk, this may take from a few minutes to several hours to complete. # mirrorcheck mirrors appear ok # # # # metastat d5: Mirror Submirror 0: d15 State: Okay Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 149512500 blocks (71 GB) d15: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACEFCd0s5 Start Block Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD 0 Dbase No State Okay August 2013 25 MPS Primary Drive Mirror Configuration Yes d25: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACFF8d0s5 Yes Start Block 0 Dbase No State Okay d6: Mirror Submirror 0: d16 State: Okay Submirror 1: d26 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 64512500 blocks (30 GB) d16: Submirror of d6 State: Okay Size: 64512500 blocks (30 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACEFCd0s6 Yes d26: Submirror of d6 State: Okay Size: 64512500 blocks (30 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACFF8d0s6 Yes Start Block 0 Start Block 0 Dbase No Dbase No State Okay State Okay d4: Mirror Submirror 0: d14 State: Okay Submirror 1: d24 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 86025000 blocks (41 GB) d14: Submirror of d4 State: Okay Size: 86025000 blocks (41 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACEFCd0s4 Yes Start Block 0 Dbase No State Okay d24: Submirror of d4 State: Okay 26 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Configuring the MPS to Mirror its Primary Drive Size: 86025000 blocks (41 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACFF8d0s4 Yes Start Block 0 Dbase No State Okay d1: Mirror Submirror 0: d11 State: Okay Submirror 1: d21 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 73412500 blocks (35 GB) d11: Submirror of d1 State: Okay Size: 73412500 blocks (35 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACEFCd0s1 Yes d21: Submirror of d1 State: Okay Size: 73412500 blocks (35 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACFF8d0s1 Yes Start Block 0 Start Block 0 Dbase No State Okay Dbase No State Okay d0: Mirror Submirror 0: d10 State: Okay Submirror 1: d20 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 10825000 blocks (5.2 GB) d10: Submirror of d0 State: Okay Size: 10825000 blocks (5.2 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACEFCd0s0 Yes d20: Submirror of d0 State: Okay Size: 10825000 blocks (5.2 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0221ACFF8d0s0 Start Block 0 Start Block Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD 0 Dbase No Okay Dbase No State State Okay August 2013 27 MPS Primary Drive Mirror Configuration Yes Device Relocation Information: Device /dev/dsk/c0t5000CCA0221ACFF8d0 /dev/dsk/c0t5000CCA0221ACEFCd0 # Reloc Yes Yes Device ID id1,sd@n5000cca0221acff8 id1,sd@n5000cca0221acefc 12. Update the crash dump facility to use the mirrrored swap meta device. Use the following command to execute dumpadm to set the crash dump device, using the output of the swap -l command to obtain the existing swap device. /usr/sbin/dumpadm -d `swap -l | tail -1 | awk '{ print $1 }'` Important: In the previous command, note that swap -l is swap + space + hyphen + lowercase letter L, tail -1 is tail + space + hyphen + the number 1, and print $1 is print + space + $ + the number 1. The command dumpadm, with no arguments, lists the crash dump status. For example: # dumpadm Dump content: kernel pages Dump device: /dev/dsk/c0t5000CCA0221ACEFCd0s1 (swap) Savecore directory: /var/crash/bl014 Savecore enabled: yes Save compressed: on # Savecore enabled: yes # dumpadm Dump content: Dump device: Savecore directory: Savecore enabled: Save compressed: # # swap -l swapfile /dev/md/dsk/d1 # 28 kernel pages /dev/md/dsk/d1 (swap) /var/crash/bl014 yes on dev 85,1 swaplo blocks free 16 73412480 73412480 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Chapter 4: Disabling Mirroring This chapter covers: 1. Configuring the MPS to Unmirror its Primary Drive Configuring the MPS to Unmirror its Primary Drive How to ... unmirror partitions 1. Make the metamirrors into a one-way mirror. See step 1 on page 29. 2. Run the metaroot command. See step 3 on page 29. 3. Edit the /etc/vfstab file. See step 3 on page 23. 4. Reboot the system. See step 5 on page 31. 5. Clear the metadevices. See step 9 on page 31. 6. Remove all entries from /etc/lvm/md.tab file. See step 6 on page 31. 7. Use the metastat command to verify that mirroring is not configured. See step 8 on page 31. 8. Update the crash dump facility to use the unmirrored swap device. See step 7 on page 31. The following example shows how to unmirror root, swap, /usr, /opt and/doc file systems. 1. Make the metamirrors into one-way mirrors by entering the following as root. 2. Determine HDD0 and the HDD0 WWN # metaroot /dev/dsk/ # diskinfo -a | grep HDD0 /SYS/HDD0 c0t5000CCA0221ACEFCd0 HITACHI H109030SESUN300G A31A # 3. Execute the metaroot command. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 29 Disabling Mirroring /dev/dsk/c0t5WWNHDD0d0s0 For example: # metaroot /dev/dsk/c0t5000CCA0221ACEFCd0s0 # The metaroot command will update slice 0 (root) in /etc/vfstab. You must execute metaroot instead of manually updating slice 0 (root) in /etc/vfstab, because root conversion requires additional special processing supplied by the metaroot command. The metaroot command will update the root slice in the /etc/vfstab file. Do not manually update the root slice in /etc/vfstab. The following example shows the /etc/vfstab file before execution of metaroot. # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no /dev/dsk/c0t5000CCA0221ACEFCd0s7 /dev/rdsk/ c0t5000CCA0221ACEFCd0s7 /mmf ufs 2 yes /dev/dsk/c0t5000CCA0221ACFF8d0s7 /dev/rdsk/ c0t5000CCA0221ACFF8d0s7 /mmf2 ufs 2 yes /devices /devices devfs no sharefs /etc/dfs/sharetab sharefs no ctfs /system/contract ctfs no objfs /system/object objfs no #swap /tmp tmpfs yes /dev/md/dsk/d1 swap no /dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 1 no /dev/md/dsk/d5 /dev/md/rdsk/d5 /opt ufs 2 yes /dev/md/dsk/d6 /dev/md/rdsk/d6 /usr ufs 1 no # The following example shows the /etc/vfstab file after execution of metaroot. # cat /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t5000CCA0221ACEFCd0s0 /dev/rdsk/ c0t5000CCA0221ACEFCd0s0 / ufs 1 no /dev/dsk/c0t5000CCA0221ACEFCd0s7 /dev/rdsk/ c0t5000CCA0221ACEFCd0s7 /mmf ufs 2 yes /dev/dsk/c0t5000CCA0221ACFF8d0s7 /dev/rdsk/ c0t5000CCA0221ACFF8d0s7 /mmf2 ufs 2 yes /devices /devices devfs no sharefs /etc/dfs/sharetab sharefs no ctfs /system/contract ctfs no objfs /system/object objfs no #swap /tmp tmpfs yes /dev/md/dsk/d1 swap no /dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 1 no /dev/md/dsk/d5 /dev/md/rdsk/d5 /opt ufs 2 yes /dev/md/dsk/d6 /dev/md/rdsk/d6 /usr ufs 1 no # 30 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Configuring the MPS to Unmirror its Primary Drive 4. Edit the /etc/vfstab file to change the following entries: /dev/md/dsk/d1 /dev/md/dsk/d4 /dev/md/dsk/d5 /dev/md/dsk/d6 /dev/md/rdsk/d4 /dev/md/rdsk/d5 /dev/md/rdsk/d6 swap /var /opt /usr ufs ufs ufs no 1 2 1 no yes no - /dev/md/rdsk/d4 /dev/md/rdsk/d5 /dev/md/rdsk/d6 swap /var /opt /usr ufs ufs ufs no 1 2 1 no yes no - to read: /dev/md/dsk/d1 /dev/md/dsk/d4 /dev/md/dsk/d5 /dev/md/dsk/d6 5. Reboot the system. # /usr/sbin/reboot 6. Remove entries from /etc/lvm/md.tab file. 7. Use the metastat command to verify that mirroring is not configured. # metastat metastat: tb019: there are no existing databases If the output indicates that there are no existing databases, then mirroring is no longer configured. 8. Update the crash dump facility to use the unmirrrored swap meta device. Use the following command to execute dumpadm to set the crash dump device, using the output of the swap -l command to obtain the existing swap device. /usr/sbin/dumpadm -d `swap -l | tail -1 | awk '{ print $1 }'` Important: In the previous command, note that swap -l is swap + space + hyphen + lowercase letter L, tail -1 is tail + space + hyphen + the number 1, print $1 is print + $ + the number 1. The command dumpadm, with no arguments, lists the crash dump status. For example: # dumpadm Dump content: Dump device: Savecore directory: Savecore enabled: Save compressed: # kernel pages /dev/dsk/c0t5000CCA0221ACEFCd0s1 (swap) /var/crash/bl014 yes on 9. Clear the metadevices. Example: # metadb -i flags first blk a m pc luo 16 c0t5000CCA0221ACEFCd0s3 a pc luo 8208 block count 8192 /dev/dsk/ 8192 /dev/dsk/ Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 31 Disabling Mirroring c0t5000CCA0221ACEFCd0s3 a pc luo 16400 8192 /dev/dsk/ c0t5000CCA0221ACEFCd0s3 a pc luo 16 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 a pc luo 8208 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 a pc luo 16400 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors # # metadb -d /dev/dsk/c0t5000CCA0221ACEFCd0s3 (HDD0) # metadb -f -d /dev/dsk/c0t5000CCA0221ACFF8d0s3 (HDD1) # Apr 4 09:46:41 T4-24-190 rpc.metad: Terminated # metadb -i # 32 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Chapter 5: Recovering from Disk Failure This chapter covers: 1. Determining the Type of Disk Failure 2. Recovering from Disk Failure If you are Able to Log In 3. Recovering from Disk Failure If you are Unable to Log In 4. If your System Fails to Boot due to a Corrupt /etc/vfstab Determining the Type of Disk Failure Disk failures, on unmirrored systems, may result in a system panic. But, when a mirrored disk fails, the Solaris Volume Manager recovers transparently, and issues console messages. The following example is taken from a mirrored system in /var/adm/messages: Aug 19 08:39:20 tb012 md_mirror: [ID 104909 kern.warning] WARNING: md: d11: /dev/dsk/ c0t5WWNHDD0d0s1 needs maintenance If the system has not rebooted and is still operational, follow the procedure If you are Able to Log into the System on page 33 to recover from disk failure. If you are unable to boot up the system, you must boot off the alternate boot device. Follow the procedure If you are Unable to Log In to the System on page 46. If you are Able to Log into the System Determine if you have a Disk or Mirror Failure Use the output of the metatstat command to determine if there is a disk or mirror failure. A state other than Resync or Active indicates that corrective action must be performed. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 33 Recovering from Disk Failure Use the procedure Recovering from a Failed Mirror Disk (Secondary Drive) on page 37 to recover from a failed mirror. Use the procedure on Recovering from a Failed Disk on page 42 to recover from a failed disk. The following sample output from the metastat command data from a working mirror. Compare your systems results to determine if and where a mirror problem may exists. root@MPSAP1-a {5} metastat d0: Mirror Submirror 0: d10 State: Okay Submirror 1: d20 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 10825000 blocks (5.2 GB) d10: Submirror of d0 State: Okay Size: 10825000 blocks (5.2 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258C2E9Cd0s0 0 No State Reloc Hot Spare Okay Yes d20: Submirror of d0 State: Okay Size: 10837500 blocks (5.2 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258B509Cd0s0 0 No State Reloc Hot Spare Okay Yes d6: Mirror Submirror 0: d16 State: Okay Submirror 1: d26 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 64512500 blocks (30 GB) d16: Submirror of d6 State: Okay Size: 64512500 blocks (30 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258C2E9Cd0s6 0 No State Reloc Hot Spare Okay Yes d26: Submirror of d6 State: Okay Size: 64525000 blocks (30 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258B509Cd0s6 0 No State Reloc Hot Spare Okay Yes d5: Mirror Submirror 0: d15 State: Okay 34 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Determine if you have a Disk or Mirror Failure Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 149512500 blocks (71 GB) d15: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258C2E9Cd0s5 0 No State Reloc Hot Spare Okay Yes d25: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258B509Cd0s5 0 No State Reloc Hot Spare Okay Yes d4: Mirror Submirror 0: d14 State: Okay Submirror 1: d24 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 86025000 blocks (41 GB) d14: Submirror of d4 State: Okay Size: 86025000 blocks (41 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258C2E9Cd0s4 0 No State Reloc Hot Spare Okay Yes d24: Submirror of d4 State: Okay Size: 86025000 blocks (41 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258B509Cd0s4 0 No State Reloc Hot Spare Okay Yes d1: Mirror Submirror 0: d11 State: Okay Submirror 1: d21 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 73737500 blocks (35 GB) d11: Submirror of d1 State: Okay Size: 73737500 blocks (35 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258C2E9Cd0s1 0 No Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD State Reloc Hot Spare Okay Yes August 2013 35 Recovering from Disk Failure d21: Submirror of d1 State: Okay Size: 73737500 blocks (35 GB) Stripe 0: Device Start Block Dbase /dev/dsk/c0t5000CCA0258B509Cd0s1 0 No State Reloc Hot Spare Okay Yes Device Relocation Information: Device Reloc Device ID /dev/dsk/c0t5000CCA0258C2E9Cd0 Yes id1,sd@n5000cca0258c2e9c /dev/dsk/c0t5000CCA0258B509Cd0 Yes id1,sd@n5000cca0258b509c root@MPSAP1-a {6} root@MPSAP1-a {6} metadb flags first blk block count a m p luo 16 8192 /dev/dsk/c0t5000CCA0258C2E9Cd0s3 a p luo 8208 8192 /dev/dsk/c0t5000CCA0258C2E9Cd0s3 a p luo 16400 8192 /dev/dsk/c0t5000CCA0258C2E9Cd0s3 a p luo 16 8192 /dev/dsk/c0t5000CCA0258B509Cd0s3 a p luo 8208 8192 /dev/dsk/c0t5000CCA0258B509Cd0s3 a p luo 16400 8192 /dev/dsk/c0t5000CCA0258B509Cd0s3 root@MPSAP1-a {7} root@MPSAP1-a {8} root@MPSAP1-a {11} Review the results to determine whether or not the hard disk requires immediate replacement. If the output of metastat shows all or the majority of submirrors residing on the same physical disk in a Needs maintenance state, this indicates a hard failure of that physical disk. It is possible that an individual slice does not report the Needs maintenance state for a considerable period of time after a disk failure. Until the OS attempts to access that slice, the Needs maintenance report may not appear. In general, if more than one slice is reported in the Needs maintenance state, assume that the disk requires replacement. If the output of metastat only shows a single submirror in a Needs maintenance state, this may indicate a recoverable momentary glitch. In this case, attempt recovery of the slice without physical disk replacement once (and only once). If a prior recovery effort already took place in the recent past and either immediately failed or failed within a few days, and the slice is again in a Needs maintenance state, replace the hard disk. Recovering from a Failed Mirror The metastat output contains the command to invoke when a submirror is reported as needs maintenance. If all mirrors contain a submirror that shows needs maintenance, then there is probably an unrecoverable hardware error, and you must replace the disk. If only one mirror contains a 36 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from a Failed Mirror Disk (Secondary Drive) submirror that shows needs maintenance, try to run the the command suggested in the metastat output. Then execute metastat to monitor the resynchronization and ensure that all states reach OK. If the state reverts back to needs maintenance, then you must replace the disk. Recovering from a Failed Mirror Disk (Secondary Drive) Use the following procedure to recover from a failed mirror disk. How to ...Recover from a Failed Mirror Disk (secondary drive) 1. Check if the State Databases are corrupted. 2. Delete corrupted databases. 3. Replace and format the failed mirror. 4. If a state database was deleted, add it back. 5. Reenable mirrors. Check the State Databases Check the State Databases using the metadb -i command. # metadb -i flags first blk block count a m p luo 16 8192 /dev/dsk/c0t5000CCA0221AD058d0s3 a p luo 8208 8192 /dev/dsk/c0t5000CCA0221AD058d0s3 a p luo 16400 8192 /dev/dsk/c0t5000CCA0221AD058d0s3 a p luo 16 8192 /dev/dsk/c0t5000CCA0221AD4ECd0s3 a p luo 8208 8192 /dev/dsk/c0t5000CCA0221AD4ECd0s3 a p luo 16400 8192 /dev/dsk/c0t5000CCA0221AD4ECd0s3 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors # Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 37 Recovering from Disk Failure Delete Corrupted State Databases If the status flags indicate a problem, use the metadb -d command to delete each corrupted database. How to ...delete corrupted State Databases 1. Use the metadb -d command to delete all stale databases. 2. Verify that all stale databases have been deleted using metadb -i. 1. Use the metadb -d command to delete all stale databases. Example: # metadb -d /dev/dsk/c0t5000CCA0221ACFF8d0s3 # #metadb -i flags first blk block count a m p luo 16 8192 c0t5000CCA0221ACEFCd0s3 a p luo 8208 8192 c0t5000CCA0221ACEFCd0s3 a p luo 16400 8192 c0t5000CCA0221ACEFCd0s3 /dev/dsk/ /dev/dsk/ /dev/dsk/ 2. Use the metadb -i command to ensure all stale databases have been deleted as shown in the previous example. Replacing a Failed Mirror How to replace a Failed Mirror 1. Repair the disk if necessary. 2. Format disk, if necessary. 3. Use metadb -a command to add back State database replicas (if replicas were corrupted.) 4. Use metareplace to reenable mirrors (if mirrors were corrupted.) Check /etc/vfstab for incorrect entries. 38 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Replacing a Failed Mirror 1. Replace the failed disk if necessary. T4-1 server HDDs are designed to be hotswapped so a complete shutdown in not required. The drive needs to be taken off line from the OS. 2. Hot-Plug - Remove Faulty HDD (Physical Disk) HDDx will be unconfigured and removed # diskinfo –a /SYS/HDD0 c0t5000CCA0258C142Cd0 HITACHI H106030SDSUN300G A2B0 /SYS/HDD1 c0t5000CCA0258BDAD8d0 HITACHI H106030SDSUN300G A2B0 # cfgadm -al | grep disk c4::w5000cca0258c142d,0 disk-path connected configured unknown c5::w5000cca0258bdad9,0 disk-path connected configured unknown Unconfigure HDDx # cfgadm -c unconfigure c5::w5000cca0258bdad9,0 # cfgadm -al | grep disk c4::w5000cca0258c142d,0 disk-path connected configured unknown c5::w5000cca0258bdad9,0 disk-path connected unconfigured unknown HDD1 LED will go blue and the drive is ready for hot-plug removal Note: If the HDD does not change state to unconfigured it may still have mounted file systems. Remove the faulty HDDx Example above shows HDD1 – Notice the SASAddress is used and NOT the SASDeviceName. 3. Hot-Plug - Install Replacement HDD (Physical Disk). A new HDDx will be installed and configured Install the replacement HDD in location HDDx # diskinfo –a /SYS/HDD0 c0t5000CCA0258C142Cd0 HITACHI /SYS/HDD1 c0t5000CCA0258BDAD8d0 HITACHI Note: new HDD WWN for the replacement disk # cfgadm -al | grep disk c4::w5000cca0258c142d,0 disk-path unknown c5::w5000cca0258bdad9,0 disk-path unknown H106030SDSUN300G A2B0 H106030SDSUN300G A2B0 connected configured connected unconfigured Note: A new HDD WWN for the replacement disk. If new HDD is in “unconfigured” state # cfgadm -c configure c5::w5000cca0258bdad9,0 Disk drive LED will go green # cfgadm -al | grep disk c4::w5000cca0258c142d,0 disk-path Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD connected configured August 2013 39 Recovering from Disk Failure unknown c5::w5000cca0258bdad9,0 unknown disk-path connected configured Example above show HDD1. 4. Configure the new disk. • Format the disk to have exactly the same size slices as the mirror. A mirror slice must be the same size or larger than the slice it is mirroring. When the primary disk and secondary mirror disk have identical size slices, either disk may assume the role of primary disk successfully. This configuration provides us the most flexible mirroring environment. In particular, when we boot from a secondary disk during a recovery effort, that former secondary disk has assumed the primary role. In order to recover the system successfully in this situation, the slices all must have identical sizes. If they are not identical, the slices on the former primary disk are too small for a resync to succeed. The recovery fails during resync with an error reporting that the mirror slice is too small. If the primary drive is repartitioned with larger slices to accommodate this limitation, upon reboot, the secondary drive is found to have slices that are too small. If you must partition opposing halves of a mirror set with mismatched partition sizes, entirely disable mirroring so as to physically retarget the secondary disk to permanently assume the primary role. Using the following procedure, recovery is possible with only one mirror restoration effort. a. Disable mirroring. (Refer to the Disabling Mirror chapter). i. Use the diskinfo -a command to determine the WWN information of both HDD0 and HDD1. ii. Use the metastat command to determine which of the two HDDs is faulted and which is functional. iii. Modify the /etc/vfstab file to now use HDD WWN address information and not metadevices. iv. Edit the /etc/system file and remove all references to mirroring. Remove all entries from the file md.tab. b. Issue the # halt command. c. Physically remove the bad HDD. d. Boot the system from the good HDD. Example: boot disk0 or boot disk1 e. Perform a ground up mirror setup. • Issue the following command on one line to install the boot record: # installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk / dev/rdsk/c0tWWNnewHDDd0s0 40 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Replacing a Failed Mirror Use the appropriate WWN HDD information in the command above. 5. Use the metadb -a command to reconfigure state database replicas. Create the initial state database and replicas: # metadb -c 3 -a -f /dev/dsk/c0tWWNofHDDd0s3 Example: # metadb -c 3 -a -f /dev/dsk/c0t5000CCA0221ACFF8d0s3 # When the first state database is created, youwill receive a message saying metadb:There are no existing databases . This is expected, as there are no databases yet created on the partition. Use the following command to create additional replicas on other disks: metadb -c 3 -a /dev/dsk/c0tWWNofHDDd0s3 6. Execute the metadb -i command to check that all six databases are created. # metadb -i flags first blk block count a m p luo 16 8192 /dev/dsk/ c0t5000CCA0221ACEFCd0s3 a p luo 8208 8192 /dev/dsk/ c0t5000CCA0221ACEFCd0s3 a p luo 16400 8192 /dev/dsk/ c0t5000CCA0221ACEFCd0s3 a u 16 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 a u 8208 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 a u 16400 8192 /dev/dsk/ c0t5000CCA0221ACFF8d0s3 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors # 7. Use metareplace to reenable the submirrors (if the submirrors were corrupted). REPLACE WITH: EXAMPLE # metareplace -e d0 /dev/dsk/c0t5000CCA0221ACFF8d0s0 d0: device c0t5000CCA0221ACFF8d0s0 is enabled # metareplace -e d1 /dev/dsk/c0t5000CCA0221ACFF8d0s1 d1: device c0t5000CCA0221ACFF8d0s1 is enabled # metareplace -e d6 /dev/dsk/c0t5000CCA0221ACFF8d0s6 d6: device c0t5000CCA0221ACFF8d0s6 is enabled # metareplace -e d4 /dev/dsk/c0t5000CCA0221ACFF8d0s4 d4: device c0t5000CCA0221ACFF8d0s4 is enabled Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 41 Recovering from Disk Failure # metareplace -e d5 /dev/dsk/c0t5000CCA0221ACFF8d0s5 d5: device c0t5000CCA0221ACFF8d0s5 is enabled # After some time, the resyncs are complete. You can now return to booting from the original device. The metastat command now displays that all submirrors are Okay. Recovering from a Failed Disk In this example, a disk that contains half of the database replicas and submirrors for root, swap, and /usr fails. Because half of the replicas are missing, the system cannot be rebooted unless it is booted into single user maintenance mode. To recover from a failed disk, you must first replace, then restore corrupted or stale metadevice databases.: Replace Corrupted or Stale Metadevice Databases How to Replace Stale Metadevice Databases 1. Delete stale replicas and reboot. 2. Repair the disk. 3. Add back all database replicas. 4. Reenable broken submirrors. 1. Delete stale replicas and reboot. Refer to Delete Corrupted State Databases on page 38 for detailed instructions. 2. Repair the disk. If your system is an architecture such as Sunfire, which supports dynamic hardware configuration, then you can remove the failed drive and insert the new drive while the system is up and running. If your system is not a Sunfire or does not support dynamic hardware configuration, you must halt the system by issuing the # halt command, replace the failed disk, and reboot. A mirror slice must be the same size or larger than the slice it is mirroring. When the primary disk and secondary mirror disk have identical size slices, either disk may assume the role of primary disk successfully. This configuration provides the most flexible mirroring environment. In particular, when you boot from a secondary disk during a recovery effort, that former secondary disk has assumed the primary role. In order to recover the system successfully in this situation, the slices all must have identical sizes. If they are not identical, the slices on the former primary disk are too small for a resync to succeed. The recovery fails during resync with an error reporting that the mirror slice is too small. If the primary drive is repartitioned with 42 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from a Failed Disk larger slices to accommodate this limitation, upon reboot, the secondary drive is found to have slices that are too small. If you must partition opposing halves of a mirror set with mismatched partition sizes, entirely disable mirroring so as to physically retarget the secondary disk to permanently assume the primary role. Using the following procedure, recovery is possible with only one mirror restoration effort. a. Disable mirroring (Refer to the Disabling Mirror chapter). i. Use the diskinfo -a command to determine the WWN information of both HDD0 and HDD1. ii. Use the metastat command to determine which of the two HDDs is faulted and which is functional. iii. Modify the /etc/vfstab file to now use HDD WWN address information and not metadevices. iv. Edit the /etc/system file and remove all references to mirroring. Remove all entries from the file md.tab. b. Issue the # halt command. c. Physically remove the bad HDD. d. Install the new blank disk in the empty c1t1 slot. e. Boot the system from the good HDD. Example: boot disk0 or boot disk1 f. Perform a ground up mirror setup. 3. Add back all database replicas. 4. Reenable broken submirrors. Restore Corrupted or Stale Metadevice Databases How to Restore Stale Metadevice Databases 1. Boot the machine to determine which replicas are down. 2. Use the metadb command to look at the database. 3. Delete the stale database replicas using the -d option to the metadb command. 4. Reboot the system. 5. When you obtain a replacement disk, halt the system, replace the failed disk, and once again, reboot the system. 6. Use the metadb command to add back the database replicas and to determine that the replicas are correct. 7. Use the metareplace command to reenable the submirrors. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 43 Recovering from Disk Failure 1. Boot the machine to determine which replicas are down. 2. Use the metadb command to look at the database. # metadb -i flags first blk block count M Wm p l o 16 unknown /dev/dsk/ c0t5000CCA0258B4C74d0s3 M W p l o 8208 unknown /dev/dsk/ c0t5000CCA0258B4C74d0s3 M W p l o 16400 unknown /dev/dsk/ c0t5000CCA0258B4C74d0s3 a m p lu 16 8192 /dev/dsk/ c0t5000CCA0258B4FA8d0s3 a p l 8208 8192 /dev/dsk/ c0t5000CCA0258B4FA8d0s3 a p l 16400 8192 /dev/dsk/ c0t5000CCA0258B4FA8d0s3 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errors # 3. Delete the stale database replicas using the -d option of the metadb command. Because, at this point, the root file system is read-only, ignore the mddb.cf error messages: # metadb -d /dev/dsk/c0t5000CCA0258B4C74d0s3 # # metadb -i flags first blk block count a m p lu 16 8192 c0t5000CCA0258B4FA8d0s3 a p l 8208 8192 44 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] /dev/dsk/ /dev/dsk/ August 2013 Recovering from a Failed Disk c0t5000CCA0258B4FA8d0s3 a p l 16400 c0t5000CCA0258B4FA8d0s3 8192 /dev/dsk/ 4. When you obtain a replacement disk, halt the system, replace and format the failed disk. Then, reboot the system again. If your system is an architecture such as Sunfire, which supports dynamic hardware configuration, then you can remove the failed drive and insert the new drive while the system is up and running. If your system is not a Sunfire or does not support dynamic hardware configuration, you must halt the system by issuing the # halt command, replace the failed disk, then reboot. A mirror slice must be the same size or larger than the slice it is mirroring. When the primary disk and secondary mirror disk have identical size slices, either disk may assume the role of primary disk successfully. This configuration provides us the most flexible mirroring environment. In particular, when we boot from a secondary disk during a recovery effort, that former secondary disk has assumed the primary role. In order to recover the system successfully in this situation, the slices all must have identical sizes. If they are not identical, the slices on the former primary disk are too small for a resync to succeed. The recovery fails during resync with an error reporting that the mirror slice is too small. If the primary drive is repartitioned with larger slices to accommodate this limitation, upon reboot, the secondary drive is found to have slices that are too small. If you must partition opposing halves of a mirror set with mismatched partition sizes, entirely disable mirroring so as to physically retarget the secondary disk to permanently assume the primary role. Using the following procedure, recovery is possible with only one mirror restoration effort. (Refer to the Disabling Mirror chapter) a. Use the diskinfo -a command to determine the WWN information of both HDD0 and HDD1. b. Use the metastat command to determine which of the two HDDs is faulted and which is functional. c. Modify the /etc/vfstab file to now use HDD WWN address information and not metadevices. d. Edit the /etc/system file and remove all references to mirroring. e. Remove all entries from the file md.tab. f. Issue the halt command. g. Physically remove the bad HDD. h. Boot the system from the good HDD. Example: boot disk0 or boot disk1 i. Perform a ground up mirror setup. Use the format command to partition the disk as it was before the failure. # format Searching for disks...done Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 45 Recovering from Disk Failure AVAILABLE DISK SELECTIONS: 0. c0t5000CCA0221ACEFCd0 solaris /scsi_vhci/disk@g5000cca0221acefc 1. c0t5000CCA0221ACFF8d0 solaris /scsi_vhci/disk@g5000cca0221acff8 Specify disk (enter its number): 5. Select the disk you wish to process 6. Install the boot block onto the raw disk root slice. Run the following command on one line: /usr/sbin/installboot /usr/platform/ `uname -i`/lib/fs/ufs/bootblk /dev/rdsk/[root-slice] For example: installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/ c0tWWNnewHDDd0s0 7. Use the metadb command to add back the database replicas and to determine that the replicas are correct. # metadb -c 3 -a /dev/dsk/c0tWWNofHDDd0s3 # metadb 8. Use the metareplace command to reenable the submirrors. # metareplace -e d0 c0tWWNofHDDd0s0 d0: device c0tWWNofHDDd0s0 is enabled # metareplace -e d1 c0tWWNofHDDd0s1 d1: device c0tWWNofHDDd0s1 is enabled # metareplace -e d4 c0tWWNofHDDd0s4 d4: device c0tWWNofHDDd0s4 is enabled # metareplace -e d5 c0tWWNofHDDd0s5 d5: device c0tWWNofHDDd0s6 is enabled # metareplace -e d6 c0tWWNofHDDd0s6 d6: device c0tWWNofHDDd0s6 is enabled The submirrors will now resync. This may take several hours. 9. After the resynchronization is complete, run metastat to check the status of the submirrors. If you are Unable to Log In to the System If your boot device fails, a message similar to the following appears. The message may differ among various architectures. 46 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from Boot Device Failure Look up the recorded information about the boot device Recording Disk Partition Information on page 8. Recovering from Boot Device Failure You need to set up an alternate boot device. The basic procedure is to repair the disk, boot from another root submirror, and then restore the database and mirrors to their original state. How to ...Recover From Boot Device Failure 1. Boot from another root submirror. 2. Use the metadb command to determine which database replicas have failed. 3. Use the metastat command to determine that half of the root, swap and /usr mirrors have failed. 4. Halt the system, repair the disk, and reboot. Note that you must reboot from the other half of the root mirror. 5. Use the metadb command to delete the failed replicas and then add them back. 6. Use the metareplace command to reenable the submirrors. In the following example, the boot device containing two of the six database replicas and the root, swap, and /usr submirrors fails. Because only two of the six database replicas failed, you can still boot. If this were not the case, you would have to delete the database replicas in singleuser mode. This procedure is described in Recovering from a Failed Disk on page 42. 1. Boot from another root submirror, using the boot command previously determined on step 7 on page 24. {0} ok boot -s disk0 or disk1 2. Use the metadb command to determine which database replicas have failed. Example( has no failures) # metadb flags first blk a m p luo 16 c0t5000CCA0221ACEFCd0s3 a p luo 8208 c0t5000CCA0221ACEFCd0s3 block count 8192 /dev/dsk/ 8192 /dev/dsk/ Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 47 Recovering from Disk Failure a p luo c0t5000CCA0221ACEFCd0s3 a p luo c0t5000CCA0221ACFF8d0s3 a p luo c0t5000CCA0221ACFF8d0s3 a p luo c0t5000CCA0221ACFF8d0s3 # 16400 8192 /dev/dsk/ 16 8192 /dev/dsk/ 8208 8192 /dev/dsk/ 16400 8192 /dev/dsk/ 3. Use the metastat command to determine which file systems failed. Example: root@MPSAP1-a {5} metastat d0: Mirror Submirror 0: d10 State: Okay Submirror 1: d20 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 10825000 blocks (5.2 GB) d10: Submirror of d0 State: Okay Size: 10825000 blocks (5.2 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258C2E9Cd0s0 Yes d20: Submirror of d0 State: Okay Size: 10837500 blocks (5.2 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258B509Cd0s0 Yes Start Block 0 Start Block 0 Dbase No Dbase No State Okay State Okay d6: Mirror Submirror 0: d16 State: Okay Submirror 1: d26 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 64512500 blocks (30 GB) d16: Submirror of d6 State: Okay Size: 64512500 blocks (30 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258C2E9Cd0s6 Yes Start Block 0 Dbase No State Okay d26: Submirror of d6 48 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from Boot Device Failure State: Okay Size: 64525000 blocks (30 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258B509Cd0s6 Yes Start Block 0 Dbase No State Okay d5: Mirror Submirror 0: d15 State: Okay Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 149512500 blocks (71 GB) d15: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258C2E9Cd0s5 Yes d25: Submirror of d5 State: Okay Size: 149512500 blocks (71 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258B509Cd0s5 Yes Start Block 0 Start Block 0 Dbase No State Okay Dbase No State Okay d4: Mirror Submirror 0: d14 State: Okay Submirror 1: d24 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 86025000 blocks (41 GB) d14: Submirror of d4 State: Okay Size: 86025000 blocks (41 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258C2E9Cd0s4 Yes d24: Submirror of d4 State: Okay Size: 86025000 blocks (41 GB) Stripe 0: Device Reloc Hot Spare Start Block 0 Start Block Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Dbase No State Okay Dbase State August 2013 49 Recovering from Disk Failure /dev/dsk/c0t5000CCA0258B509Cd0s4 Yes 0 No Okay d1: Mirror Submirror 0: d11 State: Okay Submirror 1: d21 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 73737500 blocks (35 GB) d11: Submirror of d1 State: Okay Size: 73737500 blocks (35 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258C2E9Cd0s1 Yes d21: Submirror of d1 State: Okay Size: 73737500 blocks (35 GB) Stripe 0: Device Reloc Hot Spare /dev/dsk/c0t5000CCA0258B509Cd0s1 Yes Device Relocation Information: Device /dev/dsk/c0t5000CCA0258C2E9Cd0 /dev/dsk/c0t5000CCA0258B509Cd0 root@MPSAP1-a {6} Reloc Yes Yes Start Block 0 Start Block 0 Dbase State No Okay Dbase State No Okay Device ID id1,sd@n5000cca0258c2e9c id1,sd@n5000cca0258b509c 4. Halt the system, repair the disk, and reboot. Important: You must reboot from the other half of the root mirror. # halt The "halt" command has been modified to shutdown the system gracefully. {0} ok boot -s disk0 or disk1 5. Use the metadb command to delete the failed replicas and then add them back. root@MPSAP1-a {11} metadb flags first blk a m p luo 16 c0t5000CCA0258C2E9Cd0s3 a p luo 8208 c0t5000CCA0258C2E9Cd0s3 a p luo 16400 c0t5000CCA0258C2E9Cd0s3 a p luo 16 c0t5000CCA0258B509Cd0s3 a p luo 8208 50 block count 8192 /dev/dsk/ 8192 /dev/dsk/ 8192 /dev/dsk/ 8192 /dev/dsk/ 8192 /dev/dsk/ Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from Boot Device Failure c0t5000CCA0258B509Cd0s3 a p luo 16400 c0t5000CCA0258B509Cd0s3 8192 /dev/dsk/ root@MPSAP1-a {13} metadb -d /dev/dsk/c0t5000CCA0258C2E9Cd0s3 root@MPSAP1-a {14} metadb flags first blk block count a p luo 16 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 a p luo 8208 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 a p luo 16400 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 root@MPSAP1-a {15} root@MPSAP1-a {15} metadb -c3 -a /dev/dsk/c0t5000CCA0258C2E9Cd0s3 root@MPSAP1-a {16} root@MPSAP1-a {16} metadb flags first blk block count a u 16 8192 /dev/dsk/ c0t5000CCA0258C2E9Cd0s3 a u 8208 8192 /dev/dsk/ c0t5000CCA0258C2E9Cd0s3 a u 16400 8192 /dev/dsk/ c0t5000CCA0258C2E9Cd0s3 a p luo 16 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 a p luo 8208 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 a p luo 16400 8192 /dev/dsk/ c0t5000CCA0258B509Cd0s3 root@MPSAP1-a {17} 6. Use the metareplace command to re enable the submirrors. # metareplace -e d0 c0tWWNofHDDd0s0 d0: device c0tWWNofHDDd0s0 is enabled # metareplace -e d1 c0tWWNofHDDd0s1 d1: device c0tWWNofHDDd0s1 is enabled # metareplace -e d4 c0tWWNofHDDd0s4 d4: device c0tWWNofHDDd0s4 is enabled # metareplace -e d5 c0tWWNofHDDd0s5 d5: device c0tWWNofHDDd0s6 is enabled # metareplace -e d6 c0tWWNofHDDd0s6 d6: device c0tWWNofHDDd0s6 is enabled Wait until the submirrors have re synchronized. 7. Check the mirror states by running the metastat command. # metastat or # metastat | grep sync State: Resyncing Resync in progress: 0 % done State: Resyncing /dev/dsk/c0t5000CCA0221ACFF8d0s5 Yes State: Resyncing Resync in progress: 0 % done State: Resyncing /dev/dsk/c0t5000CCA0221ACFF8d0s6 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD 0 No Resyncing 0 No Resyncing August 2013 51 Recovering from Disk Failure Yes Yes Yes Yes # State: Resyncing Resync in progress: 0 % done State: Resyncing /dev/dsk/c0t5000CCA0221ACFF8d0s4 0 No Resyncing State: Resyncing Resync in progress: 1 % done State: Resyncing /dev/dsk/c0t5000CCA0221ACFF8d0s1 0 No Resyncing State: Resyncing Resync in progress: 12 % done State: Resyncing /dev/dsk/c0t5000CCA0221ACFF8d0s0 0 No Resyncing 8. If all the submirror states are OK, reboot. Recovering from Dual Disk Failure Use your system tape back up to recover from a dual disk failure. 1. Boot from the CDROM. boot cdrom -s 2. Determine if the disks are in fact damaged. Unless there was obvious damage to the system, such as physical trauma or a power surge, a dual disk failure is likely the result of operator error. a. Try mounting the disks and interrogating the contents. b. Run fsck. If you can navigate the disk and fsck runs successfully without producing SCSI read or write errors, there is probably nothing wrong with the disks. 3. Prepare the disks to receive the tape back up. If the disks are damaged, replace them and repartition the new disks. Install new file systems on the slices with the newfs command. If the disks can be recovered, assume everything is corrupt and issue the command newfs on every slice. Assume the database slice is corrupt and run newfs on slice 3. This allows you to recreate the metadatabases as if they had never existed before. 4. Restore the most recent tape backup that was created using the mirrorbu script. 52 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 If your System Fails to Boot due to a Corrupt /etc/vfstab The file system will be setup as if in an active mirror configuration. You cannot boot from this configuration as there are no metadatabases. Follow the procedure shown in step 7 on page 71 to restore from tape. 5. Apply a boot block as shown in step 8 on page 71 6. Position the restored system to boot. a. Mount HDD0 as /mnt mount /dev/dsk/c0tWWNofHDD0d0s0 /mnt b. Edit the file /mnt/etc/system. Delete the entire section that falls in the space between the following lines: * Begin MDD root info (do not edit) * End MDD database info (do not edit) c. Remove all lines from the /mnt/etc/lvm/md.tab file. d. Restore the /mnt/etc/vfstab file to reference conventional devices. Issue the command: cp /mnt/etc/vfstab.orig /mnt/etc/vfstab The system should now boot in a fully mirror disables state. reboot the system. 7. Edit the /mnt/etc/vfstab file so that old WWN disk information is replaced with the data from the new HDD drive. The new WWN information can be found on the affixed label or use the format command to see the new WWN data. 8. Set up mirroring on the system. Begin with Allocating a New Partition (s3) on page 12. If your System Fails to Boot due to a Corrupt /etc/vfstab The vfstab can become corrupt if the metaroot command was not used to maintain mirroring changes to root, or if an old vfstab was manually copied, which does not show disk devices consistent with the metadatabases. How to ...Edit /etc/vfstab Entries 1. Run fsck and remount root read/write so you can edit the /etc/vfstab file. 2. Edit the /etc/vfstab file to contain the correct metadevice entries. 3. Reboot the machine with the reboot command. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 53 Recovering from Disk Failure If you do not make proper entries in the /etc/vfstab file, the system may fail to reboot. The following example shows how you can edit the /etc/vfstab file to recover from a failed boot. If you have failed to make the proper entry in the /etc/vfstab file when mirroring root, the machine will appear at first to be booting properly. In the following example, root is mirrored with a two-way mirror. The root entry in /etc/vfstab has somehow reverted back to the original component of the file system, but the information in /etc/system still shows the machine as booting from a metadevice. The most likely reason for this is that the metaroot command was not used to maintain /etc/system and /etc/vfstab, or an old copy of /etc/vfstab was copied back. To remedy this situation, you need to edit /etc/vfstab while in single-user mode. Correct /etc/vfstab file looks something like the following:: #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # fd /dev/fd fd no /proc /proc proc no /dev/dsk/c0t5000CCA0258C2E9Cd0s1 swap no /dev/dsk/c0t5000CCA0258C2E9Cd0s0 /dev/rdsk/ c0t5000CCA0258C2E9Cd0s0 / ufs 1 no /dev/dsk/c0t5000CCA0258C2E9Cd0s6 /dev/rdsk/c0t5000CCA0258C2E9Cd0s6 usr ufs 1 no /dev/dsk/c0t5000CCA0258C2E9Cd0s4 /dev/rdsk/c0t5000CCA0258C2E9Cd0s4 var ufs 1 no /dev/dsk/c0t5000CCA0258C2E9Cd0s7 /dev/rdsk/c0t5000CCA0258C2E9Cd0s7 mmf ufs 2 yes /dev/dsk/c0t5000CCA0258B509Cd0s7 /dev/rdsk/c0t5000CCA0258B509Cd0s7 mmf2 ufs 2 yes /dev/dsk/c0t5000CCA0258C2E9Cd0s5 /dev/rdsk/c0t5000CCA0258C2E9Cd0s5 opt ufs 2 yes /devices /devices devfs no sharefs /etc/dfs/sharetab sharefs no ctfs /system/contract ctfs no objfs /system/object objfs no #swap /tmp tmpfs yes # / / / / / Because of the errors, the machine automatically goes into single-user mode when booted: Caution: Do not type Ctrl-d. At this point, root and /usr are mounted as read-only. Follow these steps to recover from a failed boot: 1. Run fsck and remount root read/write so you can edit the /etc/vfstab file. Caution: Be careful to use the correct metadevice for root. 54 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 If your System Fails to Boot due to a Corrupt /etc/vfstab 2. Edit the /etc/vfstab file to contain the correct metadevice entries. Edit the root entry in the /etc/vfstab file as shown in the following example: #device device mount #to mount to fsck point # fd /dev/fd fd /proc /proc proc /dev/dsk/c0t5000CCA0258C2E9Cd0s1 no /dev/dsk/c0t5000CCA0258C2E9Cd0s0 c0t5000CCA0258C2E9Cd0s0 / /dev/dsk/c0t5000CCA0258C2E9Cd0s6 c0t5000CCA0258C2E9Cd0s6 /usr /dev/dsk/c0t5000CCA0258C2E9Cd0s4 c0t5000CCA0258C2E9Cd0s4 /var /dev/dsk/c0t5000CCA0258C2E9Cd0s7 c0t5000CCA0258C2E9Cd0s7 /mmf /dev/dsk/c0t5000CCA0258B509Cd0s7 c0t5000CCA0258B509Cd0s7 /mmf2 /dev/dsk/c0t5000CCA0258C2E9Cd0s5 c0t5000CCA0258C2E9Cd0s5 /opt /devices /devices sharefs /etc/dfs/sharetab ctfs /system/contract objfs /system/object objfs #swap /tmp tmpfs # FS type no no - fsck pass - /dev/rdsk/ ufs 1 /dev/rdsk/ ufs 1 /dev/rdsk/ ufs 1 /dev/rdsk/ ufs 2 /dev/rdsk/ ufs 2 /dev/rdsk/ ufs 2 devfs sharefs ctfs no yes - mount mount at boot options swap - no - no - no - yes - yes - yes no no no - - - 3. Reboot the machine with the reboot command. # reboot Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 55 Recovering from Disk Failure 56 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 mirrorbu Script Overview The mirrorbu information contained in this appendix is also available as a separate document on the Avaya LiveLink documentation site. The mirrorbu script available at /etc/mirrorbu is used to back up mirrored systems. The mdsk package installs the mirrorbu and the associated scripts during the manufacturing of Avaya mirrored systems. Earlier systems may contain an older version of mirrorbu not related to mdsk or without mirrorbu at all. Refer to mdsk Package Processing on page 75 to verify that mdsk has been installed on your system before proceeding. Using the mirrorbu Script 1. Mirrorbu can only be run on a system that has active, synchronized mirrors managed by Solaris Volume Manager. 2. Mirrorbu depends on a system disk partition scheme equal to systems manufactured at Avaya. Slices 0,1, and 3 are expected to be root, swap and metadata. 3. Each file system (except root) will be write-locked while one of its mirrors is taken offline. Anything that attempts to write to the locked file system will hang until it is unlocked, so run mirrobu during a time when the system is least active. After the mirror is offline, the file system is unlocked (write allowable). The file system will not be mirrored while it is offline. 4. Run ufsdump to dump the offlined mirror to tape, and optionally, another disk. File systems are dumped consecutively to the tape. If the tape becomes full, you are prompted to insert another tape. If you wish to place each file system dump on a separate tape, specify the -o option on the mirrorbu commandline. You will be prompted for a new tape for every file system when the -o option is specified. 5. Version 1.3 of mdsk or higher supports remote tape drives. If the tape drive is on a remote system, use the mirrorbu -t option and prefix the tape device with the hostname, or IP address, and a colon (for example, tapeserver:/dev/rmt/0). The remote system must grant rsh permissions to the system executing the command. Permissions are granted by configuring /.rhosts or /etc/hosts.equiv. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 57 6. Version 1.4 of mdsk or higher is required for Solaris 10. Version 1.4 of mdsk also includes an optional additional backup to a disk. The -d option specifies the disk target backup device. The disk target backup device must be dedicated to backup and cannot used for any other purpose. The optional -b option is used with the d option to make the backup target disk bootable from its present location. 7. The mirror is brought back online when the ufsdump is complete, and mirror re synchronization automatically occurs. 8. The back up procedure takes one to several hours, depending on the amount of data being backed up. 9. Mirrorbu dumps the mirrored slices on the system drive to tape. Back up the mmf file systems and non-mirrored file systems separately. 10. The optional disk backup formats the backup disk according to the system disk vtoc and copies all slices from the system disk to the backup disk. Existing data on the backup disk is destroyed. 11. Run the following commands if you are operating vi from an ASCII terminal instead of a graphics console: • # TERM=sun; export TERM • # EDITOR=vi; export EDITOR If the command TERM=sun does not supply a usable screen, you may need to use TERM=vt100. 12. If mirrorbu is invoked by cron, be sure to check the cron log in /var/log and the mirrorbu log in /etc/mirrorBUdata. Also check mail to the root user. The ufsdump will fail if it requires a tape change when invoked by cron. 13. The Solaris Volume Manager metacommands are found in the default PATH. Preparing to Use mirrorbu Prepare to use mirrorbu by first recording data you need later and then unmirror the system in preparation for backup. 1. Log in as root. (Execute su -) 2. Make note of the partition table sizes of the system disk and the mirror disk so it can be reproduced on a new disk. 3. Take note of the mirror disk boot path or target, in case you need to boot off the mirror during recovery. The mirror is on the same controller as the system disk so you can specify disk[x] as the drive to boot where [x] is the target number. 4. Record the alternate boot path, in case you will later need to boot off the mirror during recovery. Refer to Primary Boot Disk Failure on page 65 for options on 58 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Preparing to Use mirrorbu pointing to an alternate boot path in the boot command at the OK prompt. The alternate boot path is the device link of the boot mirror, so note the output of: # diskinfo -a Enclosure path: Chassis Serial Number: Chassis Model: Label ---------/SYS/HDD0 /SYS/HDD1 1308BD0044-physical-hba-1 1308BD0044-physical-hba-1 ORCL,Netra-SPARC-T4-1 Disk name ---------------------c0t5000CCA0221ACEFCd0 c0t5000CCA0221ACFF8d0 Vendor -------HITACHI HITACHI Product ---------------H109030SESUN300G H109030SESUN300G Vers ---A31A A31A # ls -la /dev/dsk/c0t5000CCA0221ACFF8d0s0 2 lrwxrwxrwx 1 root root 48 Jul 16 11:21 /dev/dsk/ c0t5000CCA0221ACFF8d0s0 -> ../../devices/scsi_vhci/disk@g5000cca0221ac # # ls -la /dev/dsk/c0t5000CCA0221ACEFCd0s0 2 lrwxrwxrwx 1 root root 48 Jul 16 11:21 /dev/dsk/ c0t5000CCA0221ACEFCd0s0 -> ../../devices/scsi_vhci/disk@g5000cca0221ac # You can verify which target is the alternate boot by viewing md.tab and searching for the line beginning with /dev/md/dsk/d20 1 1. In the line below, /dev/dsk/c0tWWNd0s0 is the root mirror for the system disk. /dev/ md/dsk/d20 1 1 /dev/dsk/c0tWWNd0s0 md.tab resides in /etc/lvm. 5. Choose a backup time when the system is least active. Because the mirror will be taken offline and dumped, it is not necessary to terminate applications, however, there will be a small window when the file system is locked that can cause an application write to hang. The file system is locked while the mirror is taken offline, then immediately unlocked. The offline mirror is then dumped and the mirror is put online. Extensive writes to the unmirrored primary slice may generate a small performance degradation when the mirror is put online and forced to re synchronize. 6. Determine the number of tapes required. To determine how many tapes you will require, execute df -k and add up the usage for all the displayed system disk slices. The space allocation listed under the used heading will be in Kbytes when the -k option is used. All system disk slices shown in the df command output must be backed up, with the exception of swap. Swap does not need to be backed up because it contains temporary data. 7. Check the state of the Solaris Volume Manager. Execute the following command. # /etc/mirrorcheck Proceed with one of the following actions, based on the mirrorcheck results: • If the output shows that the mirrors appear ok, proceed to Running mirrorbu on page 61. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 59 • If the output shows that the states are not ok, continue with step 8. 8. Review /tmp/mirrorcheck.out. If all states are OK, the mirrors may be configured but not attached. For example: # /etc/mirrorcheck /dev/dsk/c0tWWNd0s0 /dev/dsk/c0tWWNd0s1 /dev/dsk/c0tWWNd0s4 /dev/dsk/c0tWWNd0s5 /dev/dsk/c0tWWNd0s6 a. Check /tmp/mirrorcheck.out for more information. b. Here, the mirrors have been configured but not attached, and the mirror configuration during system creation has not been completed. /etc/ mirrorvps exists at this point, and it renames itself to /etc/mirrorvps.done when it is complete. To complete the mirror configuration do the following: c. Run # /etc/mirrorvps. d. When /etc/mirrorvps script is complete, it will rename itself to /etc/ mirrorvps.done. The mirrors will now synchronize, and you must wait until the synchronization is complete. Monitor synchronization activity with the metastat command. For example: # metastat | grep sync State: Resync State: State: Resync State: State: Resync State: State: Resync State: State: Resync State: Resyncing in progress: Resyncing Resyncing in progress: Resyncing Resyncing in progress: Resyncing Resyncing in progress: Resyncing Resyncing in progress: Resyncing 33 % done 7 % done 20 % done 2 % done 9 % done Intermittently execute the above metastat command until grep finds no match for sync. e. To verify that all stats are ok, run the metastat command. # metastat All states should indicate OK and be attached. For example, d5: Mirror Submirror 0: d15 60 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Running mirrorbu State: Okay Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 7811244 blocks f. The following is the metastat output of a mirror metadevice that has been created but not yet mirrored (submirrors not attached). Note that only one submirror is shown attached to mirror d5. Usually, two submirrors would appear. d5: Mirror Submirror 0: d15 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 7811244 blocks d15: Submirror of d5 State: Okay Size: 7811244 blocks Stripe 0: Device Start Block Dbase State c0t3d0s5 . . d25: Concat/Stripe Size: 7816248 blocks Stripe 0: Device Start Block Dbase c0t2d0s5 Hot Spare All detached slices will appear at the bottom of the metastat output. g. To attach d25 to the d5 mirror metadevice, run the following command: # metattach /dev/md/dsk/d5 /dev/md/dsk/d25 h. Edit /etc/vfstab so that /dev/dsk/c0t3d0s5 and /dev/rdsk/c0t3d0s5 are /dev/md/dsk/d5 and /dev/md/rdsk/d5 (if not already done). Running mirrorbu 1. Change directory to /etc. # cd /etc 2. Run mirrorbu, specifying commandline options when necessary. The mirrorbu command line options are as follows: Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 61 -b Make the backup disk bootable from the target location where it presently resides. This option is useful when the -d option is also specified. This option is ignored if d is not specified. -d Create an unmirrored copy of the system disk on the specified disk, where is of the form c0tWWNd0. For example, mirrorbu -d c0t5000CCA0258B4FA8d0 is formatted like the system disk, therefore, all existing data on is destroyed. -o One slice per tape. If this option is specified, you will be prompted to insert a new backup tape for each slice. If this option is not specified, dump files will be stacked on the same tape. -t Tape device. Do not append a trailing n for norewind; this is handled by the script. If the -t option is not specified, /dev/rmt/0 is used by default. For example, # ./mirrorbu (to use the default /dev/rmt/0) or # ./mirrorbu -t /dev/rmt/1 (to indicate the tape is on /dev/rmt/1) A remote tape device may be used, with mirrorbu from mdsk version 1.3 or later. The system to be backed up must have rsh permissions granted by the system with the tape drive. Permissions can be achieved by configuring /.rhosts or /etc/ hosts.equiv on the system with the tape drive. For example, if mirrorbu is run from a system with hostname mps1a, then /.rhosts on the tape server would contain the line: mps1a root For example, using tape drive on remote system tb019: # ./mirrorbu -t tb019:/dev/rmt/0 to indicate the tape is on remote host tb019 /dev/rmt/0. For example, using tape drive on remote system tb019, additionally creating an unmirrored copy of the system disk on c0tWWNd0 (An unused 3rd HDD installed in the HDD2 drive bay), and making c0tWWNd0 (HDD2) bootable from its target location: # ./mirrorbu -t tb019:/dev/rmt/0 -bd c0tWWNd0 (HDD2) 3. You will receive a message with the log name. When the backup is complete, watch the screen output and check the log for problems or errors. Compare your output to the following sample: # mirrorbu ====================================================== The log for this backup is /etc/mirrorBUdata/mirrorbu.out041002-17:04:41 ====================================================== This script will backup filesystems by taking the mirror offline and dumping it. Reduncancy will be lost for the duration of the backup and regained when the mirror resynchs - after the backup. note the size of /dev/md/dsk filesystems, you may require more than one tape. Filesystems will be dumped in this order: 62 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Running mirrorbu Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d0 481067 51671 381290 12% / /dev/md/dsk/d4 770187 9018 722660 2% /var /dev/md/dsk/d5 3843668 528061 3123424 15% /opt /dev/md/dsk/d6 1488162 723320 705316 51% /usr taking / /dev/md/dsk/d20 offline from /dev/md/dsk/d0 Dumping / /dev/md/rdsk/d20.......please wait...... DUMP: Writing 63 Kilobyte records DUMP: Date of this level 0 dump: Wed Apr 10 17:04:48 2002 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/md/rdsk/d20 to /dev/rmt/0n. DUMP: Mapping (Pass I) [regular files] DUMP: Mapping (Pass II) [directories] DUMP: Estimated 109936 blocks (53.68MB). DUMP: Dumping (Pass III) [directories] DUMP: Dumping (Pass IV) [regular files] DUMP: 109870 blocks (53.65MB) on 1 volume at 731 KB/sec DUMP: DUMP IS DONE bringing / /dev/md/dsk/d20 back online to /dev/md/dsk/d0 Locking /var taking /var /dev/md/dsk/d24 offline from /dev/md/dsk/d4 unlocking unmirrored /var Dumping /var /dev/md/rdsk/d24.......please wait...... DUMP: Writing 63 Kilobyte records DUMP: Date of this level 0 dump: Wed Apr 10 17:06:09 2002 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/md/rdsk/d24 to /dev/rmt/0n. DUMP: Mapping (Pass I) [regular files] DUMP: Mapping (Pass II) [directories] DUMP: Estimated 25432 blocks (12.42MB). DUMP: Dumping (Pass III) [directories] DUMP: Dumping (Pass IV) [regular files] DUMP: 25324 blocks (12.37MB) on 1 volume at 913 KB/sec DUMP: DUMP IS DONE bringing /var /dev/md/dsk/d24 back online to /dev/md/dsk/d4 Locking /opt taking /opt /dev/md/dsk/d25 offline from /dev/md/dsk/d5 unlocking unmirrored /opt Dumping /opt /dev/md/rdsk/d25.......please wait...... DUMP: Writing 63 Kilobyte records DUMP: Date of this level 0 dump: Wed Apr 10 17:06:30 2002 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/md/rdsk/d25 to /dev/rmt/0n. DUMP: Mapping (Pass I) [regular files] DUMP: Mapping (Pass II) [directories] DUMP: Estimated 1057522 blocks (516.37MB). DUMP: Dumping (Pass III) [directories] DUMP: Dumping (Pass IV) [regular files] DUMP: 82.28% done, finished in 0:02 DUMP: 1057390 blocks (516.30MB) on 1 volume at 747 KB/sec DUMP: DUMP IS DONE bringing /opt /dev/md/dsk/d25 back online to /dev/md/dsk/d5 Locking /usr taking /usr /dev/md/dsk/d26 offline from /dev/md/dsk/d6 unlocking unmirrored /usr Dumping /usr /dev/md/rdsk/d26.......please wait...... DUMP: Writing 63 Kilobyte records DUMP: Date of this level 0 dump: Wed Apr 10 17:18:32 2002 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/md/rdsk/d26 to /dev/rmt/0n. DUMP: Mapping (Pass I) [regular files] DUMP: Mapping (Pass II) [directories] DUMP: Estimated 1497140 blocks (731.03MB). DUMP: Dumping (Pass III) [directories] DUMP: Dumping (Pass IV) [regular files] Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 63 DUMP: 51.17% done, finished in 0:09 DUMP: 1497130 blocks (731.02MB) on 1 volume at 637 KB/sec DUMP: DUMP IS DONE bringing /usr /dev/md/dsk/d26 back online to /dev/md/dsk/d6 Backup done....rewinding..... Sample log output associated with the above screen: taking / /dev/md/dsk/d20 offline from /dev/md/dsk/d0 d0: submirror d20 is offlined Did metaoffline /dev/md/dsk/d0 /dev/md/dsk/d20 Dumping / /dev/md/rdsk/d20.......please wait...... Did /usr/sbin/ufsdump 0cf /dev/rmt/0n /dev/md/rdsk/d20 bringing / /dev/md/dsk/d20 back online to /dev/md/dsk/d0 d0: submirror d20 is onlined Did metaonline /dev/md/dsk/d0 /dev/md/dsk/d20 Locking /var Did /usr/sbin/lockfs -w /var taking /var /dev/md/dsk/d24 offline from /dev/md/dsk/d4 d4: submirror d24 is offlined Did metaoffline /dev/md/dsk/d4 /dev/md/dsk/d24 unlocking unmirrored /var Did /usr/sbin/lockfs -u /var Dumping /var /dev/md/rdsk/d24.......please wait...... Did /usr/sbin/ufsdump 0cf /dev/rmt/0n /dev/md/rdsk/d24 bringing /var /dev/md/dsk/d24 back online to /dev/md/dsk/d4 d4: submirror d24 is onlined Did metaonline /dev/md/dsk/d4 /dev/md/dsk/d24 Locking /opt Did /usr/sbin/lockfs -w /opt taking /opt /dev/md/dsk/d25 offline from /dev/md/dsk/d5 d5: submirror d25 is offlined Did metaoffline /dev/md/dsk/d5 /dev/md/dsk/d25 unlocking unmirrored /opt Did /usr/sbin/lockfs -u /opt Dumping /opt /dev/md/rdsk/d25.......please wait...... Did /usr/sbin/ufsdump 0cf /dev/rmt/0n /dev/md/rdsk/d25 bringing /opt /dev/md/dsk/d25 back online to /dev/md/dsk/d5 d5: submirror d25 is onlined Did metaonline /dev/md/dsk/d5 /dev/md/dsk/d25 Locking /usr Did /usr/sbin/lockfs -w /usr taking /usr /dev/md/dsk/d26 offline from /dev/md/dsk/d6 d6: submirror d26 is offlined Did metaoffline /dev/md/dsk/d6 /dev/md/dsk/d26 unlocking unmirrored /usr Did /usr/sbin/lockfs -u /usr Dumping /usr /dev/md/rdsk/d26.......please wait...... Did /usr/sbin/ufsdump 0cf /dev/rmt/0n /dev/md/rdsk/d26 bringing /usr /dev/md/dsk/d26 back online to /dev/md/dsk/d6 d6: submirror d26 is onlined Did metaonline /dev/md/dsk/d6 /dev/md/dsk/d26 Backup done....rewinding..... 4. If the following message is received, then the current tape is full. You must eject the tape, insert a new tape, and then respond yes to the following prompt. DUMP: NEEDS ATTENTION: Is the new volume (#2) 64 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering from Disk Failure mounted on `SYSTEMNAME:/dev/rmt/0n' and ready to go?: ("yes" or "no") yes 5. After the tape rewinds, eject it and write the date, sequence number, and number of tapes in this backup set on the label. For example, write: • [today's date] mirrorbu 1 of 1 (when everything fit on a single tape). • or [today's date] mirrorbu 1 of 2 (on first tape) and [today's date] mirrorbu 2 of 2 (on second tape). • Write the file system order on the external tape label. This can help identify the correct file number when restoring. The mirrorbu screen output, as well as the mirrorbu.out file corresponding to this dump, shows the order of file systems on the dump tape. Recovering from Disk Failure If you must replace one disk, try to acquire the same disk geometry type as the other half of the mirror. If the disks have different geometry, you may need to allocate slightly larger slice sizes on the new disk to compensate for size rounding factors. Refer to Recovering from a Failed Disk on page 42 and the Solaris Volume Manager Administration Guide for more info. Some restore scenarios are described below. Primary Boot Disk Failure Here the boot device can be any of the following: not responding, cannot be opened, or has stale or insufficient meta databases. If disk mirroring has recovered your system and it is not down, there is no need to boot at this point. Proceed with the metadb commands in step 5 to update mirroring so as not to use the databases on the failed disk. Netra T4-1 servers have the mirrored HDDs on the same controller. In fact all four HDD drive slots HDD0 - HDD3 are on the same controller. In the example below the mirrored HDD pair are at HDD0 and HDD1. The normal boot disk would be HDD0 (disk0). 1. At the OK prompt, run boot disk0 -s (to boot from the primary disk HDD0)or boot disk1 -s (to boot from the secondary disk HDD1). 2. Press Enter. 3. If you cannot boot onto the mirror, it is possible that the corruption on the system drive has been mirrored onto the submirror from which you are trying to boot. Therefore, you must stop following this procedure and refer to section Recovering when All Drives in the System Disk Mirror Fail on page 70. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 65 4. If you receive one of the following messages, the system may still be recoverable. Continue with step 5 in this procedure: • Insufficient metadevice database replicas located • Use metadb to delete databases which are broken • Ignore any "Read-only filesystem" error messages 5. Although root is not writable, you must remove the failing disk's metadatabases from operation. The metadatabases must be deleted from the failing disk, and the system rebooted, before you replace the disk. a. Display the metadatabases with the metadb command: # metadb -i b. Delete the metadatabases from the indicated disk. The metadatabases reside in slice 3. The following command deletes the metadatabases from the boot disk c0tWWNHDD0d0. # metadb -d -f c0tWWNd0s3 Ignore any read-only file system messages. Although root is read-only, the metadatabases in slice 3 have been updated. c. If the metadb -i output shows the mirror disk metadatabases are unknown, and the boot disk metadatabases are usable, then delete the databases from the mirror disk, and proceed with the steps to recover the mirror disk at Secondary Boot Disk (Mirror) Failure on page 70. 6. Run the halt command. If the halt command yields the root prompt (#), then you need to utilize the ILOM to get to the ok prompt. 7. Boot the system so that root is adjusted to the updated mirror status. boot disk1 –s Messages about mirrors requiring maintenance appear during boot. 8. Detach the submirrors located on the system disk (which you are about to replace). # /etc/mirrordetachboot The metadetach commands in this script yield messages indicating that re synchronization is in progress. You must continue with the following steps to complete the mirror replacement. 9. Run the halt command. 10. Verify that you can boot onto the mirror without affecting the system disk: boot disk1 -s 11. Replace the failed disk. 66 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Primary Boot Disk Failure a. If the system is an architecture that supports dynamic hardware configuration, such as Sunfire, replace the failed drive (for example, c0tWWNHDD0d0). b. If the system is not an architecture that supports dynamic hardware configuration, such as Sunfire, then halt and power down before replacing the disk, then boot onto the mirror as done previously (for example, boot disk1 -s). 12. Execute the prtvtoc command to see if the disk geometries match. The geometry is listed under the dimensions heading. If the dimensions do not match, you will need to move the mirror drive to the system SCSI target, and place the new drive into the mirrored SCSI target. Swapping the SCSI target is necessary when the disk geometry differs because you need to allocate the new slices slightly larger than the mirrored slices, due to format rounding factors, to recover the data. If you preserve the original SCSI targets, you will get boot failure after recovery because it cannot mirror from the larger recovered partitions to the smaller original partitions. The following is an example of equal geometry: # diskinfo -a Enclosure path: Chassis Serial Number: Chassis Model: Label ---------/SYS/HDD0 /SYS/HDD1 /SYS/HDD2 1223BD0589-physical-hba-1 1223BD0589-physical-hba-1 ORCL,Netra-SPARC-T4-1 Disk name ---------------------c0t5000CCA0258C2E9Cd0 c0t5000CCA0258B509Cd0 c0t5000CCA0258B4FA8d0 Vendor -------HITACHI HITACHI HITACHI Product ---------------H106030SDSUN300G H106030SDSUN300G H106030SDSUN300G Vers ---A2B0 A2B0 A2B0 # prtvtoc /dev/rdsk/c0t5000CCA0258C2E9Cd0s2 * /dev/rdsk/c0t5000CCA0258C2E9Cd0s2 (volume "solaris") partition map * * Dimensions: * 512 bytes/sector * 625 sectors/track * 20 tracks/cylinder * 12500 sectors/cylinder * 46875 cylinders * 46873 accessible cylinders * * Flags: * 1: unmountable * 10: read-only * ... # prtvtoc /dev/rdsk/c0t5000CCA0258B509Cd0s2 * /dev/rdsk/c0t5000CCA0258B509Cd0s2 (volume "solaris") partition map * * Dimensions: * 512 bytes/sector * 625 sectors/track * 20 tracks/cylinder * 12500 sectors/cylinder * 46875 cylinders * 46873 accessible cylinders * Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 67 * Flags: * 1: unmountable * 10: read-only * ... If the disk geometries match, skip to Format the Failed Drive Slices on page 68. If the disk geometries differ, do the following: a. Revert the root mirror to a regular file. # metaroot /dev/dsk/c0tWWNd0s0 b. Edit /mnt/etc/system and comment out all the lines between: * Begin MDD root info (do not edit) * End MDD database info (do not edit) Important: Comments in the /etc/system file begins with *. c. Run the command: # mv /mnt/etc/vfstab.orig /mnt/etc/vfstab d. Halt, power down, and swap targets. For example, rejumper the mirrored disk to target 3, and the new disk to target 2. Install into the newly jumpered SCSI locations and boot. e. Format the new drive to have mirror slice sizes the at least the same size as what is now the primary drive. f. Run newfs command on the newly created partitions. g. Skip the remaining steps here and proceed with the step following Uncomment the Lines in /etc/system on page 72. Edit the /etc/system file after reboot. Format the Failed Drive Slices 1. Format the failed drive slices exactly as shown in the partition tables saved by you during Preparing to Use mirrorbu on page 58. Refer to Partitioning Guidelines on page 76 for partitioning considerations. 2. Run newfs for the newly created partitions. 3. Add the metadatabases to slice 3 of the replaced disk: # metadb -a -c 3 /dev/dsk/c0tWWNd0s3 . 4. Determine the actions required by running the metastat command and viewing the output. At this point, submirrors should have the following lines in the metastat output: State: needs maintenance Invoke [command] 68 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Primary Boot Disk Failure Where [command] could be an example of metareplace or metasync. Mirrorreplace, Metareplace, Metasync, and Metastat 1. If the invoke lines all show metareplace commands, execute the supplied mirrorreplaceboot script to replace them all. (If you are referred to this step from Secondary Boot Disk (Mirror) Failure on page 70, then substitute mirrorreplace instead of mirrorreplaceboot). # /etc/mirrorreplaceboot System messages appear telling you that the re synchronization is in progress. 2. Recheck the status of the submirrors and execute the appropriate command per the invoke line. a. Run the command # metastat | more. The output of the metastat command displays the next command to run: • metastat output: Invoke metareplace d6 c0tWWNd0s6 Run: metareplace -e d6 c0tWWNd0s6 Run the metareplace to replace the existing target with the new at the same target through the -e option. • metastat output: Invoke metasync d6 Run: metasync d6 Run metasync as shown. b. Check the resulting status: #metastat | more c. Repeat this step until all mirrors show State: OK or State:Synching. 3. Run the following command to check metastat and wait until all mirrors have moved from State: resynching to State: OK. # metastat | more Install Boot Block 1. Install the boot block onto the raw disk root slice of the disk you just replaced. Run the following command on one line: installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/ c0tWWNnewHDDd0s0 Example :# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/ c0t5000CCA0258BDAD8d0s0 2. When all mirrors show state: OK, you may boot off the system disk: # reboot Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 69 Secondary Boot Disk (Mirror) Failure After secondary boot disk (mirror) failure, the mirror yields messages about the metadevices on the mirror disk needing maintenance or replacement, or metadatabase errors. 1. Verify the health of the meta databases, and delete them from the disk you are about to replace, as done in step 5 of Primary Boot Disk Failure on page 65. 2. Look at metastat output to determine the next steps: # metastat 3. If metastat indicates the drive needs replacement (all submirrors on the drive show needing replacement), then do the following before proceeding. a. Detach the mirror: # /etc/mirrordetach b. If the system is not an architecture, such as Sunfire, that supports dynamic hardware configuration, halt and powerdown. c. Replace the disk, format as shown in the partition tables you saved during Preparing to Use mirrorbu on page 58. d. Run newfs on all new slices except slice 2 (backup partition) and slice 3, which contain the metadatabases. 4. If metadatabases were deleted in step 1, add the metadatabases to slice 3 of the mirror disk: # metadb -a -c 3 /dev/dsk/c0tWWNd0s3 5. Determine the required actions as shown in the metastat output. Follow steps 17 to 19 in Primary Boot Disk Failure on page 65. Use the mirrorreplace command instead of mirrorreplaceboot. Also be sure to substitute the proper target disk. Recovering when All Drives in the System Disk Mirror Fail 1. Follow the steps Mirrorreplace, Metareplace, Metasync, and Metastat on page 69. Use the mirrorreplace command instead of mirrorreplaceboot. Also be sure to substitute the proper target disk. 2. Replace the bad drives. Refer to Additional Information on page 75. 70 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering when All Drives in the System Disk Mirror Fail 3. If a tape drive is attached to the system, proceed to step 4. If the tape is on a remote system, verify the network environment for the cdrom boot access to the remote tape. a. If the tape drive is on a remote system, and the system to be recovered is not already defined in the network, you must define it by adding an entry in the remote tape system files /etc/hosts and /etc/ethers. b. Create an entry in the remote tape system /.rhosts file. Refer to step 3 in Running mirrorbu on page 61 for more information on /.rhosts. 4. Boot Solaris from CD. Run the following command at the OK prompt: boot cdrom -s 5. Partition the drives with the same sizes as shown in the format listings you saved in Preparing to Use mirrorbu on page 58. The mirror slices must be the same size or greater than the system drive slices. 6. Run newfs on all partitions except slice 2 (backup partition) and slice 3, which will contain the metadatabases. 7. Mount each file system onto /mnt and restore the respective dump file from tape. Substitute the appropriate WWN target information as necessary. a. # mount /dev/dsk/c0tWWNd0s0 /mnt b. # cd /mnt c. If tape is remote: # ufsretore rfs :/dev/ rmt/0n 1 If tape is local: # ufsrestore rfs /dev/rmt/0n 1 The n is for norewind and the 1 skips to the next dump on tape. d. # rm restoresymtable e. # cd / f. # umount /mnt g. Restore the data on all the slices: /, /var, /opt, and /usr. Refer to Restoring Data on page 74 Important: /mmf is not backed up by the mirrorbu script and must be obtained from a separate tape. For more information on loading all the backed up data onto the system drive, refer to Restoring Data on page 74. 8. Now install the boot block onto the raw disk root slice of both drives. Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 71 Run the following command on one line for each root slice: /usr/sbin/ installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/[root-slice] For example: installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/ c0tWWNnewHDDd0s0 9. Position the restored system to boot. At this point, everything is positioned except the metadatabases. You cannot boot the restored system without them. Therefore, you must temporarily disable mirroring from processing it's configuration. For this example, c0tWWNd0s0 is the root slice to boot. Substitute your root target, if different: a. # mount /dev/dsk/c0tWWNd0s0/mnt b. Edit /mnt/etc/system and comment out all the lines between: * Begin MDD root info (do not edit) * End MDD database info (do not edit) Important: Comment lines in /etc/system begin with asterisk (*). 10. # mv /mnt/etc/vfstab.orig /mnt/etc/vfstab 11. Reboot. Uncomment the Lines in /etc/system 1. Create the metadatabases and initialize the mirrors. The mirrorcfg script that was executed during jumpstart to create and initialize mirrors may be used. This script expects no mirrors and no metadatabases to reside on the system. If you have reached this step, there should be no meta databases or mirrors on your system. / etc/mirrorcfg was renamed to /etc/mirrorcfg.done upon successful completion. # mv /etc/mirrorcfg.done /etc/mirrorcfg # /etc/mirrorcfg Be sure to correct any errors before continuing. The mirrorcfg script should not be run when partial mirrors or metadatabases exist. The mirrorcfg script can be used for examples of certain commands that can be used for individual mirror maintenance. Refer to the MPS Command Reference Manual for more information. 2. Reboot. (The restored system reboots onto metadevices with disk mirroring active but with mirrors not yet attached). 3. Attach and synchronize the mirrors: # mv /etc/mirrorvps.done /etc/mirrorvps # /etc/mirrorvps The following is a sample output from mirrorvps: # mirrorvps (Example Data) d1: submirror d21 is attached d0: submirror d20 is attached d4: submirror d24 is attached d5: submirror d25 is attached d6: submirror d26 is attached disks mirrored SUCCESSFULLY!!!! 72 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Recovering when All Drives in the System Disk Mirror Fail Alternate boot path via mirror: lrwxrwxrwx 1 root root 48 Jan 4 16:29 /dev/dsk/ c0t5000CCA0258B4FA8d0s0 -> ../../devices/scsi_vhci/ disk@g5000cca0258b4fa8:a Updating crash dump Dump content: Dump device: Savecore directory: Savecore enabled: Save compressed: # to use the dump device as a mirrored volume kernel pages /dev/md/dsk/d1 (swap) /var/crash/tb012 yes on a. The mirrors synchronize and you must wait until the synchronization is complete. Monitor synchronization activity with the metastat command. For example: # metastat | grep sync State: Resyncing Resync in progress: 33 % done State: Resyncing State: Resyncing Resync in progress: 7 % done State: Resyncing State: Resyncing Resync in progress: 20 % done State: Resyncing State: Resyncing Resync in progress: 2 % done State: Resyncing State: Resyncing Resync in progress: 9 % done State: Resyncing Intermittently execute the above metastat command until grep finds no match for sync. b. Verify all states are OK. # metastat All states should indicate OK and be attached. Metastat output example of attached mirror: d5: Mirror Submirror 0: d15 State: Okay Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 7811244 blocks Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 73 Restoring Data Use ufsrestore to load files from tape onto the current directory. The directory must be a stable file system. Data on disk is overwritten, so make sure you are in the directory you want to restore. The following is an example of restoring an entire ufsdump file from a dump tape, /dev/rmt/0, to /opt. If the tape is on a remote system, prefix the tape device with the hostname, or IP address, and a colon, for instance tapeserver:/dev/rmt/0. The remote system must grant rsh permissions to the system sending the ufsrestore command. Permissions can be achieved by configuring /.rhosts or /etc/hosts.equiv 1. Change to the directory you want to restore, for example, /opt. Then run the following command: # cd /opt If you are presently booted from the CD, then you should mount the disk slice you want to restore onto /mnt, then cd to /mnt. For example: # mount /dev/dsk/ c0tWWNd0s5 /mnt # cd /mnt 2. Make sure your tape is positioned at the beginning, so it can skip to the location you intend. Run: # mt -f /dev/rmt/0 rew If tape is on a remote host named tapeserver: # rsh tapeserver # mt -f /dev/rmt/0 rew # exit 3. If you are not sure whether /opt is the third dump tape on the file, you can list the file's table of contents on the dump tape to see if it resembles /opt. The norewind flag is required on the tape device. Run: # ufsrestore tfs /dev/rmt/0n 3 If tape is on a remote host named tapeserver: # ufsrestore tfs tapeserver:/dev/rmt/ 0n 3 4. Rewind to the beginning of the tape. Run: # mt -f /dev/rmt/0 rew If tape is on a remote host named tapeserver: # rsh tapepserver # mt -f /dev/rmt/0 rew # exit 5. Execute ufsrestore to skip to the third file (if you know that /opt has been backed up to the third file on your dump tape). Note: The norewind flag is required on the tape device. Run: 74 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013 Additional Information # ufsrestore rfs /dev/rmt/0n 3 If tape is on a remote host named tapeserver: # ufsrestore rfs tapeserver:/dev/rmt/ 0n 3 6. When the restore is complete, remove the file: # rm restoresymtable 7. Rewind the tape: # mt -f /dev/rmt/0 rew If tape is on a remote host named tapeserver: # rsh tapepserver # mt -f /dev/rmt/0 rew # exit 8. Eject the tape. Additional Information mdsk Package Processing 1. Verify that mdsk version 1.4 or higher has been installed. Note the second line of the pkginfo -x command output. # pkginfo -x mdsk mdsk scripts for disksuite bu and maint (SPARC) mdsk1.4 Example of mdsk not installed: # pkginfo -x mdsk ERROR: information for "mdsk" was not found 2. Check that the package is installed. # pkginfo mdsk system mdsk scripts for disksuite bu and maint Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD August 2013 75 Partitioning Guidelines A mirror slice must be the same size or larger than the slice it is mirroring. When the primary disk and secondary mirror disk have identical size slices, either disk may assume the role of primary disk successfully. When booting from the secondary disk during a recovery effort, the former secondary disk has assumed the primary roll. In order to recover the system successfully in this situation, the slices all must have identical sizes. If they are not identical, the slices on the former primary disk will be too small for a resync to succeed. The recovery will fail during resync with an error reporting that the mirror slice is too small. If the primary drive is re partitioned with larger slices to accommodate this limitation, upon reboot, the secondary drive will be found to have slices that are too small. If you must partition opposing halves of a mirror set with mismatched partition sizes, you must disable mirroring, physically replace the primary boot disk with the secondary boot disk, install the new disk as the secondary disk, and configure for mirroring. Refer to this guide for instructions. 76 Avaya MPS Disk Mirroring Configuration and Recovery with WWN Type HDD Comments? [email protected] August 2013