Transcript
Exchange Server 2013 Solution Reviewed Program (ESRP)
EMC VNX5400 UNIFIED STORAGE 20,000 USERS VIRTUALIZED MICROSOFT EXCHANGE SERVER 2013 MAILBOX RESILIENCY STORAGE SOLUTION Tested with: ESRP – Storage Version 4.0 Test date: March 2014
EMC SOLUTIONS Abstract This white paper describes the technical validation of a 20,000-user Exchange 2013 storage solution deployed on EMC® VNX5400™ unified storage and virtualized on Microsoft Hyper-V according to criteria specified by the Microsoft Exchange Solution Reviewed Program (ESRP) – Storage program. The performance results and best practices presented in this paper provide validated guidelines for configuring the VNX5400 storage system for a large enterprise Exchange Server 2013 environment. March 2014
Copyright © 2014 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Part Number H12997
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
2
Table of contents Overview .............................................................................................................................................. 5 Disclaimer ............................................................................................................................................ 5 Features ............................................................................................................................................... 6 EMC VNX series of unified storage platforms .................................................................................... 6 Architecture................................................................................................................................. 6 VNX software ............................................................................................................................... 7 VNX FAST VP ................................................................................................................................ 8 EMC Virtual Provisioning technology on VNX ............................................................................... 9 EMC VNX5400 specifications....................................................................................................... 9 Solution description ........................................................................................................................... 11 Storage design for Exchange Server 2013 ...................................................................................... 12 Building block used in this solution ............................................................................................... 13 VNX storage pools and data protection .......................................................................................... 14 Mailbox server scalability factors ................................................................................................... 15 Exchange virtualization .................................................................................................................. 15 Targeted customer profile .............................................................................................................. 15 Tested deployment ........................................................................................................................ 16 Best practices..................................................................................................................................... 19 Hyper-V virtual Fibre Channel ......................................................................................................... 19 Mailbox server virtual machines ..................................................................................................... 19 Hypervisor host .............................................................................................................................. 20 Networking .................................................................................................................................... 20 Core storage................................................................................................................................... 20 Backup strategy ............................................................................................................................. 21 Information resources ........................................................................................................................ 21 Test results summary ......................................................................................................................... 22 Storage reliability (stress test) ....................................................................................................... 22 Storage performance...................................................................................................................... 22 Individual server metrics ........................................................................................................... 22 Performance across servers ....................................................................................................... 23 Database backup and recovery performance.................................................................................. 23 Database read-only performance............................................................................................... 24 Transaction log recovery/replay performance ............................................................................ 24
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
3
Detailed test results ........................................................................................................................... 24 How to view Jetstress reports ......................................................................................................... 24 Conclusion ......................................................................................................................................... 25 Contact EMC .................................................................................................................................. 25
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
4
Overview This document provides information about EMC® VNX5400™ unified storage array performance with 20,000 Microsoft Exchange Server 2013 users. Performance validation is based on Microsoft Exchange Solution Reviewed Program (ESRP)— Storage program guidelines 1. For any questions or comments regarding the content of this document, see Contact EMC on page 25.
Disclaimer This document has been produced independently of Microsoft Corporation. Microsoft Corporation expressly disclaims responsibility for, and makes no warranty, express or implied, with respect to, the accuracy of the contents of this document. The information contained in this document represents the current view of EMC on the issues discussed as of the date of publication. Due to changing market conditions, it should not be interpreted as a commitment on the part of EMC. In addition, EMC cannot guarantee the accuracy of any information presented after the date of publication.
1
The ESRP—Storage program was developed by Microsoft Corporation to provide a common storage testing framework for vendors to provide information on their storage solutions for Microsoft Exchange Server software. For more details on the Microsoft ESRP—Storage program, refer to http://technet.microsoft.com/en-us/exchange/ff182054.aspx.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
5
Features EMC VNX series of unified storage platforms
The EMC® VNX™ series delivers industry-leading innovation and enterprise capabilities for file, block, and object storage in a scalable, easy-to-use solution. This next-generation storage platform, shown in Figure 1, combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today’s enterprises.
Figure 1.
EMC VNX series of unified storage platforms
EMC VNX series unified storage systems deliver uncompromising scalability and flexibility for the mid-tier while providing market-leading simplicity and efficiency to minimize total cost of ownership. Architecture Based on the powerful new family of Intel Xeon E5-2600 (Sandy Bridge) processors, the EMC VNX implements a modular architecture that integrates hardware components for block, file, and object with concurrent support for native networkattached storage (NAS), Internet small computer system interface (iSCSI), Fibre Channel (FC), and Fibre Channel over Ethernet (FCoE) protocols. The series delivers file (NAS) functionality via two to eight X-Blade data movers and block (iSCSI, FCoE, and FC) storage via dual storage processors (SPs) using full 6 Gb/s Serial Attached SCSI (SAS) disk drive topology. The system includes the patented EMC MCx™ multicore storage software operating environment that delivers unparalleled performance efficiency. You can start with block or file functionality and easily upgrade to unified when needed. The unified configuration includes the following rack-mounted enclosures: •
Block Services: Disk processor enclosure (includes disk drives) or SP enclosure (no drives included) plus standby power system. Capacity for block or file use cases is added via disk-array enclosures (DAEs).
•
File and Unified Services: One or more data mover enclosures and a control station to deliver file protocols.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
6
Customers can benefit from the new VNX features such as 2: •
Next-generation unified storage, optimized for virtualized applications
•
Extended cache using flash drives with EMC Fully Automated Storage Tiering (FAST™) Cache and FAST for Virtual Pools (FAST VP) that can be optimized for the highest system performance and lowest storage cost simultaneously on both block and file
•
Multiprotocol support for file, block, and object, with object access through EMC Atmos® Virtual Edition (Atmos VE)
•
Simplified management with EMC Unisphere™, which provides a single management framework for all NAS, SAN, and replication needs
•
Up to three times faster performance with the latest Intel multicore CPUs optimized for flash
•
6 Gb/s SAS back end supporting the latest drive technologies including a variety of 2.5-inch and 3.5-inch drive types such as flash, SAS, and nearline SAS (NL-SAS), of varying capacity sizes
•
Expanded EMC UltraFlex™ I/O connectivity—FC, iSCSI, Common Internet File System (CIFS), Network File System (NFS) including parallel NFS (pNFS), MultiPath File System (MPFS), and FCoE connectivity for converged networking over Ethernet
VNX software The VNX series includes six software suites and two software packs, making it easier and simpler to attain the maximum overall benefits 3. All VNX software is managed through Unisphere.
Software suites The VNX software is also available in modular suites: •
FAST Suite—Automatically optimizes performance and cost to simultaneously provide the highest system performance and the lowest storage cost
•
Security and Compliance Suite—Keeps data safe from changes, deletions, and malicious activity
•
Local Protection Suite—Practices safe data protection and repurposing
•
Remote Protection Suite—Protects data against localized failures, outages, and disasters
2
Features listed are based on the VNX Operating Environment version available at the time of this solution validation. EMC constantly improves and updates its storage fleet with new features and functionalities. For the latest features and updates visit www.emc.com. 3 Features of Suites and Packs might vary depending on the system. See the EMC VNX Series Software Suites data sheet for details.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
7
•
Application Protection Suite—Automates application copies and proves compliance
•
Unisphere Management Suite—Monitors and manages multiple VNX systems with full visibility across the virtual stack
Software packs The VNX series software is also available in two comprehensive packages to ensure that customers have all of the necessary capabilities to protect and manage their information. •
VNX Total Protection Pack—Includes local, remote, and application protection suites
•
VNX Total Efficiency Pack—Includes the suites in the VNX Total Protection Pack, along with the FAST and Security & Compliance suites
For additional details about the EMC VNX series of unified storage systems, refer to the following white papers available at www.emc.com: •
Introduction to the New EMC VNX Series—VNX5200, VNX5400, VNX5600, VNX5800, VNX7600, & VNX8000
•
Introduction to the EMC VNX Series—VNX5100, VNX5300, VNX5500, VNX5700, & VNX7500
VNX FAST VP The FAST VP feature, which is included in the VNX FAST Suite, can lower the total cost of ownership (TCO) and increase performance by intelligently managing data placement according to activity level. When FAST VP is implemented, the storage system measures, analyzes, and implements a dynamic storage-tiering policy much faster and more efficiently than a human analyst could ever achieve. Storage provisioning can be repetitive and time-consuming and, when estimates are calculated incorrectly, it can produce uncertain results. It is not always obvious how to match capacity to the performance requirements of a workload’s data. Even when a match is achieved, requirements change and a storage system’s provisioning requires constant adjustments. Storage tiering allows a storage pool to use drives of varying levels of performance. Logical unit numbers (LUNs) use the storage capacity needed from the pool on the EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
8
devices with the required performance characteristics. FAST VP uses I/O statistics at a 256 MB slice granularity (known as sub-LUN tiering). The relative activity level of each slice is used to determine the need to promote to higher tiers of storage. Relocation is initiated at the user’s discretion through either manual initiation or an automated scheduler. FAST VP removes the need for manual, resource-intensive LUN migrations while still providing the performance levels required by the most active dataset. EMC Virtual Provisioning technology on VNX EMC Virtual Provisioning™ technology on VNX series systems provides pool-based storage provisioning by implementing pool LUNs that can be either thin or thick. Thin LUNs provide on-demand storage that maximizes the utilization of your storage by allocating it as it is needed. Thick LUNs provide high performance and predictable performance for your applications. Both types of LUNs benefit from the ease-of-use features of pool-based provisioning. Pools and pool LUNs are also the building blocks for advanced data services such as FAST VP and compression. Virtual Provisioning technology supports features such as hot sparing, proactive sparing, and the ability to migrate data between thin LUNs, thick LUNs, or traditional LUNs without incurring application downtime. The ability to nondisruptively migrate data to different LUN and disk types provides the best solution for meeting your changing application and business requirements without downtime. This flexibility separates Virtual Provisioning from typical thin provisioning implementations. EMC VNX5400 specifications The EMC VNX5400 model is a member of the VNX series next-generation storage platform, powered by two Intel quad-core Xeon E5-2600 1.8 GHz series processors and a 6 Gb/s SAS drive back end, providing the industry’s highest bandwidth. The EMC VNX5400 model is designed to deliver maximum performance and scalability for enterprises. It is a converged platform that replaces the EMC CLARiiON® and EMC Celerra® platforms and enables organizations to dynamically grow, share, and cost effectively manage multiprotocol file systems and multiprotocol block storage access. The VNX Operating Environment enables Microsoft Windows and Linux/UNIX clients to share files in multiprotocol (NFS and CIFS) environments. At the same time, it supports iSCSI, FC, and FCoE access for high-bandwidth and latency-sensitive block applications. Table 1 lists VNX5400 features for block storage. For additional VNX specifications for both block and file storage, see EMC VNX Series Unified Storage Systems at www.emc.com. Table 1.
VNX5400 features summary (shown for block storage only)
System feature
Value/Description
Number of drives per array
4 minimum, 250 maximum
Drive types
Flash, SAS, NL-SAS
DAE options
25 x 2.5-in. SAS/flash drives—2U 15 x 3.5-in. SAS/flash drives—3U
RAID options
0, 1, 3, 5, 6, 1/0
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
9
System feature
Value/Description
Number of SPs
2
Block protocols supported
FC, iSCSI, FCoE
CPU/memory per array
2 quad-core Intel Xeon E5-2600 @ 1.8 GHz/32 GB (16 GB for each SP)
Maximum block UltraFlex I/O modules per array
8
Maximum FAST Cache
1 TB
Maximum raw capacity
750 TB
Maximum number of SAN hosts
1,024
Maximum number of pools
15
Maximum number of LUNs (pool)
1,000
Maximum number of LUNs (classic)
2,048
Maximum number of ports per array
36
2/4/8 Gb/s FC maximum number of ports per array
32
GbE iSCSI maximum number of ports per array
16
10 GbE iSCSI maximum number of ports per array
16
FCoE maximum number of ports per array
16
6 Gb/s SAS maximum number of buses for DAE connections per array
2 x 4 lane
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
10
Solution description This solution is intended for medium-to-large enterprises that are planning to deploy and virtualize Microsoft Exchange Server 2013 on Microsoft Hyper-V and EMC storage. It uses the Exchange Server 2013 Database Availability Group (DAG) feature to provide mailbox resiliency and high availability for Exchange users. The solution design represents a virtualized Exchange Server 2013 environment supporting 20,000 users in a mailbox resiliency configuration across two VNX5400 storage arrays. The solution is designed to provide outstanding performance and flexibility for today’s and tomorrow’s Exchange users. This solution employs virtualization to effectively use hardware resources that benefit the Exchange application. In this solution, the physical hypervisor server hosts multiple Exchange mailbox server virtual machines. In this solution, 20,000 users are distributed across eight mailbox servers in a DAG. The DAG has two RAID-protected copies of every Exchange database—a primary (active) copy and a secondary (passive) copy—that are evenly split between hypervisor hosts and storage pools on two VNX5400 arrays. Each Exchange database replicates to an alternate mailbox server that resides on a different Hyper-V host through the use of the Exchange native DAG host-based log shipping mechanism. Each mailbox server virtual machine is configured to support up to 5,000 users (2,500 active and 2,500 passive) with a 2 GB mailbox capacity and 0.10 I/O operations per second (IOPS) per user (including an additional 20 percent I/O headroom). This user profile corresponds to about 150 messages per user per day. The solution is designed to eliminate a single point of failure and can handle the loss of an array, hypervisor server, mailbox server virtual machine, database volume, host bus adapter (HBA), or switch. For example, during a mailbox server failure or maintenance operation, each mailbox server can handle the compute and storage requirements of all active and passive databases. Similarly, during a hypervisor host failure or maintenance activities, another host can support all mailbox servers (20,000 users with four virtual mailbox servers per hypervisor host). Note: You can compare this solution architecture to one in which Exchange Server 2013 is deployed in a stand-alone configuration (without DAG) and a single VNX5400 storage array with eight mailbox servers provides service to all 20,000 users.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
11
Figure 2 illustrates the architecture of this solution.
Figure 2.
Storage design for Exchange Server 2013
EMC VNX5400 storage solution architecture for 20,000 Exchange 2013 users
Storage design is an important element for ensuring the successful deployment of Microsoft Exchange Server 2013. Sizing and configuring storage for use with Exchange Server 2013 can be a complicated process driven by many variables and requirements that vary from organization to organization. Properly configured Exchange storage, combined with optimally sized server and network infrastructures, can guarantee a smooth Exchange operation. Defining a unit of measure—a mailbox server building block—is one method that simplifies the sizing and configuration of large amounts of storage on the EMC VNX series storage arrays for use with Exchange Server 2013. A mailbox server building block represents the amount of disk, network, and server resources required to support a specific number of Exchange Server 2013 users. The amount of required resources is derived from a specific user profile type, mailbox size, and availability requirements. Using the building block approach simplifies the design and implementation of Exchange Server 2013. Once the initial building block is designed, it can be easily reproduced to support the required number of users in your enterprise. By using this approach, EMC customers can now create their own building blocks that are based on their company’s specific Exchange environment requirements. This approach is helpful when future growth is expected because it makes Exchange environment expansion simple and straightforward. EMC best practices involving the building block approach for Exchange Server design have proven to be highly successful in many customer implementations. EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
12
Building block used in this solution
The mailbox server building block used in this solution supports up to 5,000 users per server with a 2 GB mailbox size and 0.10 IOPS per user (0.12 IOPS were validated, which reflects Microsoft's recommended overhead of 20 percent). This building block is made up of 14 x 3 TB NL-SAS drives (12 for databases and 2 for logs) and supports I/O and capacity for a single mailbox server, which in turn supports 5,000 users (2,500 active and 2,500 passive). The solution uses four of these building blocks (56 disks—48 disks for databases and 8 disks for logs) on the primary storage array to scale the configuration up to 20,000 users. The four building blocks are then duplicated on the secondary storage array. Multiple building blocks are grouped together in storage pools. In this solution we grouped two building blocks in a RAID 1/0 storage pool with 24 x 3 TB NL-SAS disks, thus supporting database I/O and mailbox capacity for two mailbox server virtual machines. We also added 2 x 100 GB flash disks to the pool for FAST VP to allow for absorption of metadata. For transactional logs, we configured separate RAID 1/0 pools with four 3 TB NL-SAS disks, each supporting two building blocks. See Figure 3 on page 14 for more details. Table 2 summarizes the attributes of the building block. Table 2.
Mailbox server building block attributes
Item
Value/Description
Number of users supported by a single mailbox server
5,000 (2,500 active and 2,500 passive)
User profile
0.10 IOPS with a 20% headroom (0.12 IOPS tested)
Mailbox size
2 GB
Drive type and capacity
3 TB NL-SAS drives; 7,200 rpm; 100 GB flash
RAID type
RAID 1/0
Database LUN size
1,300 GB
Log LUN size
55 GB
Number of disks to support database and logs for a single mailbox server
14 (12 for databases, 2 for logs)
Mailbox server virtual machine requirements
6 vCPUs, 92 GB RAM, 200 GB OS volume, 2 NICs (1 or 10 Gb/s Ethernet)*
* For mailbox server CPU and memory requirements, use the Exchange 2013 Server Role Requirements Calculator on Microsoft TechNet.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
13
VNX storage pools and data protection
In this solution, we used thin-provisioned pool-based LUNs on the VNX5400 array for the Exchange databases and logs. Using storage pools on VNX systems simplifies storage provisioning. Traditional storage provisioning with only RAID groups restricts the number of disks in a group to 16. Storage pools, on the other hand, enable you to manage potentially hundreds of disks at a time. Such pool-based provisioning provides benefits similar to metaLUN striping across many drives, but, unlike metaLUNs, storage pools require minimal planning and management effort. Storage pools support the same RAID protection levels as RAID groups do: RAID 5, RAID 6, and RAID 1/0. By using storage pools, you can take advantage of FAST VP, which can lower the total cost of ownership (TCO) and increase performance by intelligently managing data placement according to activity level. Also, using storage pools provides the capability to use VNX Snapshots for data replication and protection. EMC AppSync® uses VNX Snapshots replication technology to provide fully automated applicationconsistent replicas of Exchange data. To learn more about protecting Exchange data with EMC AppSync and VNX Snapshots, review the white paper titled Advanced Protection for Microsoft Exchange 2010 on EMC VNX Storage available at www.emc.com. Figure 3 illustrates the VNX5400 storage solution using building block methodology, designed to provide the best performance and sufficient capacity for virtualized Exchange 2013 with 20,000 users at 2 GB mailbox capacity.
Figure 3.
Exchange storage design using EMC building-block methodology
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
14
Mailbox server scalability factors
The ESRP—Storage program focuses on storage solution testing to address performance and reliability issues with storage design. However, storage is not the only factor that must be considered when designing a scalable Exchange solution. Other factors that can affect server scalability are: •
Server processor utilization
•
Server physical and virtual memory limitations
•
Resource requirements for other applications
•
Directory and network service latencies
•
Network infrastructure limitations
•
Replication and recovery requirements
•
Client usage profiles
Any combination of these factors can affect the total number of mailboxes supported by a single mailbox server. Because all of these factors fall outside the scope of ESRP—Storage program solution validation, a single mailbox server might not necessarily support 5,000 users (as validated for the solution described in this document) in every customer deployment. Exchange virtualization
Microsoft supports Exchange Server 2013 in production on hardware virtualization software such as Microsoft Hyper-V, VMware vSphere, and any third-party hypervisor that has been validated under the Windows Server Virtualization Validation Program. If Exchange mailbox servers are deployed as virtual machines, the number of users that can be supported by each mailbox server virtual machine depends on the hypervisor type, physical server hardware capabilities such as CPU and memory characteristics, and high availability requirements. For more information about Exchange Server 2013 system requirements for hardware virtualization, refer to Exchange 2013 Virtualization on Microsoft TechNet. For more information on identifying and addressing performance bottlenecks in an Exchange system, refer to Troubleshooting Microsoft Exchange Server Performance on Microsoft TechNet.
Targeted customer profile
This solution is designed for any medium-to-large enterprise that is planning to consolidate and virtualize its Exchange Server 2013 environment, and requires a highly reliable, scalable storage solution. The design of this solution, which supports 20,000 Exchange users, has the following characteristics: •
20,000 active mailboxes
•
0.10 IOPS per user (150 messages sent and received per user/day, with an additional 20 percent overhead—0.12 IOPS validated)
•
2 GB mailbox size
•
Eight Exchange servers per DAG, each supporting 5,000 users (2,500 active and 2,500 passive)
•
Native DAG replication mechanism for mailbox resiliency and high availability with two database copies maintained EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
15
•
24x7 background database maintenance (BDM) enabled
•
Two EMC VNX5400 storage arrays (the solution was validated with four mailbox servers connected to a single storage array)
Note: If you require more than two database copies, EMC recommends placing two RAIDprotected DAG copies on one storage array and placing a third RAID-protected copy on another array.
Tested deployment Table 3 through Table 8 summarize the testing environment. Table 3.
Simulated Exchange configuration
Attribute
Value
Number of Exchange mailboxes
20,000
Number of DAGs
1
Number of copies/database
2
Number of servers/DAG
8
Number of active mailboxes/server
5,000
Number of databases/mailbox server
10
Number of mailboxes/database
500
User profile
0.10 IOPS with a 20% headroom (0.12 IOPS validated)
Database LUN size
1,300 GB
Log LUN size
55 GB
Total database size for performance testing
41,360 GB (40.4 TB) (10,340 GB per mailbox server)
Percentage of storage capacity used by Exchange database 4
79.5% (40.4 TB/50.8 TB) Total database size/Total formatted capacity
4
Storage performance characteristics change based on the percentage utilization of the individual disks. Tests that use a small percentage of the storage (~25 percent) might exhibit reduced throughput if the storage capacity utilization is significantly increased beyond what is reported in this paper.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
16
Table 4.
Storage hardware
Component
Description
Storage connectivity
FCoE
Storage model and OS/firmware revision
EMC VNX5400, VNX Block Operating Environment version 05.33.000.5.038 Windows Hardware Compatibility List
Storage cache
32 GB mirrored
Number of storage processors
2
Number of storage ports used
4 (2 per SP)
Maximum bandwidth of storage connectivity to host
20 Gb/s (2 x 10 Gb/s FCoE ports)
Switch type/model/firmware revision
Cisco Nexus 5548UP/NXOS 6.0(2)N2(3)
HBA model/firmware
Cisco VIC 1240/Firmware 2.1(3b)
Number of HBAs per host
4 (2 for boot and 2 for virtual Fibre Channel)
Hypervisor server specification
Cisco Systems Inc. UCSB-B200-M3 with 2 Intel Xeon CPU E5-2660 v2 @ 2.20 GHz, 20 Core, 256 GB RAM
Number of disks used in this solution as validated
56 (48 for databases and 8 for logs)
Maximum number of disks supported by this array model
250
Table 5.
UCS Fabric Interconnect 6248/UCS Firmware 2.1(3b)
Storage software
Component
Description
HBA driver
2.3.0.20
HBA Queue Depth setting
Default
Multipathing
EMC PowerPath® (installed on Hyper-V host and guest VM)
Hypervisor type
Microsoft Windows Server 2012 R2 Hyper-V
Exchange Server virtual machine OS
Windows Server 2012 R2 Datacenter (6.3.9600)
ESE.dll file version
15.00.0847.030
Replication solution name/version
N/A
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
17
Table 6.
Storage disk configuration (mailbox store disks) 5
Attribute
Description
Disk type and speed
3 TB NL-SAS 7,200 rpm
Raw capacity for each disk
2,751 GB
Number of physical disks in test
48
Total raw storage capacity
132,048 GB (129 TB)
RAID level
RAID 1/0
Total formatted capacity
52,000 GB (50.78 TB) (13 TB per server)
Storage capacity utilization (percentage)
40.3% (50.78 TB/129 TB) Formatted capacity/total raw capacity
Database capacity utilization (percentage) Table 7.
31.3% (40.4 TB/129 TB) Total database size/total raw capacity
Storage disk configuration (transactional log disks)
Attribute
Description
Disk type and speed
3 TB NL-SAS 7,200 rpm
Raw capacity for each disk
2,751 GB
Number of physical disks in test
8
Total raw storage capacity
22,008 GB
RAID level
RAID 1/0
Total formatted capacity
2,200 GB (550 GB per server)
Table 8.
Replication Configuration
Replication mechanism
Exchange 2013 DAG mailbox resiliency
Number of links
2
Simulated link distance
LAN
Link type
IP
Link bandwidth
10 Gb/s Ethernet (converged)
5
Capacity of drives, and therefore the raw data capacity of the disk array, based on binary representation of 1KB = 1024 bytes. This means that 1 GB = 1,073,741,824 bytes.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
18
Best practices Microsoft Exchange Server 2013 has changed significantly since the early versions of Exchange, particularly with regard to I/O and storage. Many changes have been made to the core schema and the Extensible Storage Engine (ESE) to reduce the I/O footprint. Due to this I/O reduction, Exchange Server 2013 can be deployed on low-cost SATA and NL-SAS disks in addition to FC, SAS, and flash. Depending on the Exchange Server 2013 deployment model you choose, DAG or stand-alone, storage configuration options can vary from highly reliable SAN or SCSI infrastructure to direct-attached storage (DAS), and even to non-RAID JBOD configurations in specific circumstances. All RAID types are supported to meet the requirements of the highavailability model and storage configuration you choose to deploy. EMC provides storage options for all of these configurations. •
For Exchange Server 2013 storage design best practices, refer to Exchange 2013 Storage Configuration Options on Microsoft TechNet.
•
For Exchange Server 2013 sizing and capacity planning, refer to Exchange 2013 Sizing and Capacity Planning on Microsoft TechNet.
In addition to following Microsoft’s recommendations, EMC recommends that you follow the best practices described in this section to improve EMC VNX series storage performance with Exchange Server 2013. Hyper-V virtual Fibre Channel
In this solution we used virtual FC within the guest OS on the Hyper-V server. The virtual FC feature in Hyper-V on Windows Server 2012 and Windows Server 2012 R2 provides virtual FC ports within the guest operating system, which allows you to connect to virtual FC directly from within virtual machines. Therefore, a virtual machine can connect directly to a SAN through a virtualized FC adapter. For more information, requirements, and best practices about Hyper-V virtual FC, refer to Hyper-V Virtual Fibre Channel Overview on Microsoft TechNet.
Mailbox server virtual machines
Follow these recommendations to ensure the best possible mailbox server performance: •
Format Windows Server NTFS volumes to be used for Exchange databases and logs with an allocation unit size of 64 KB. Note: Partition alignment is no longer required when running Microsoft Windows Server 2008/2008 R2 or Windows Server 2012/2012 R2, because partitions are automatically set to a 1 MB alignment offset.
•
Apply all required OS hotfixes recommended by Microsoft, EMC, the hypervisor vendor, and the HBA vendor.
•
Reserve CPU and memory resources required for each mailbox server virtual machine.
•
In virtual environments, install virtual machine tools to optimize performance.
•
Configure two network adapters, one for the management network and one for the DAG replication network. For DAG replication, 10 Gb/s virtual network adapters are more appropriate. EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
19
Hypervisor host
Networking
To configure the hypervisor host for optimal Exchange performance follow these recommendations: •
In SAN environments, use redundant HBAs connected to different fabrics.
•
Verify that the HBA installed in the server can support the IOPS requirements, even in a failover situation. To avoid throttling, ensure that the queue depth is set according to EMC recommendations for the specific HBA vendor and type.
•
Install EMC PowerPath for optimal path management and maximum I/O performance. For more information on installing and configuring PowerPath, visit www.emc.com/storage/powerpath/powerpath-multipathing.htm.
•
Depending on your backup solution, configure Exchange disks as virtual disks or SCSI pass-through disks.
•
Apply all required hypervisor OS hotfixes recommended by Microsoft, EMC, the hypervisor vendor, and the HBA vendor.
For high availability deployments that use DAGs, we recommend multiple physical NICs connected to different networks to isolate user MAPI traffic from database replication traffic. For iSCSI deployments, we prefer multiple network switches for fault tolerance and performance. Where this is not possible, use VLANs to isolate iSCSI traffic.
Core storage
EMC VNX5400 is already optimized for Exchange Server 2013 workloads and requires only a few minor adjustments. Follow these guidelines to ensure optimal storage performance: •
Isolate the Microsoft Exchange server database workload from other I/O-intensive applications or workloads (assign the Exchange workload to its own set of disks). This ensures the highest level of performance for Exchange and simplifies troubleshooting in the event of a disk-related performance issue.
•
For VNX models that preceded next-generation systems (VNX5100, VNX5300, VNX5500, VNX 5700, VNX7500), set the VNX storage array page size parameter to 16 KB, and allocate 1 GB of available VNX array SP memory to read cache and the remaining available memory to write cache. Note: On next-generation VNX systems such as the VNX5400 model, you do not have to set the page size and the memory for read and write cache. These settings are now managed automatically.
•
Ensure that write cache is enabled for all database and log LUNs.
•
Plan for performance even in a failover situation. Ensure SP utilization does not exceed 70–75 percent.
•
Ensure that I/O load and bandwidth are evenly balanced and distributed across the VNX5400 front-end ports and back-end buses for failover and load balancing.
•
When sizing storage, always prioritize IOPS over capacity. After calculating the IOPS requirements, assess the capacity that is required. EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
20
•
After establishing the required IOPS, apply a 20 percent I/O overhead factor to build in adequate reserve throughput capability.
•
Consider additional throughput requirements imposed by background database maintenance (BDM). Exchange 2013 BDM is approximately 2.25 MB/s per database.
•
For most Exchange deployments, use storage pools made up of disks of the same type with the appropriate RAID multiplier. In configurations with high IOPS requirements, using multitiered pools (FAST VP) might be beneficial.
•
Ensure that each DAG copy is isolated in a different storage pool.
•
Isolate the databases from the logs on separate sets of disks for optimal performance.
For additional information about configuring VNX storage for best performance, review the white paper titled EMC VNX Unified Best Practices for Performance available on www.emc.com. For a complete list of EMC storage best practices for Microsoft Exchange, as well as using EMC FAST VP, EMC FAST Cache, and EMC XtremCache™ technologies with Exchange workloads, review the white paper titled Microsoft Exchange Server Best Practices and Design Guidelines on EMC Storage available on www.emc.com. Backup strategy
When you are using intelligent VNX storage arrays for Exchange storage, we recommend using EMC AppSync to take application-consistent Volume Shadow Copy Service (VSS) snapshots to protect your Exchange data.
Information resources For more information on EMC solutions for Microsoft Exchange Server, visit www.emc.com/exchange.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
21
Test results summary This section provides a high-level summary of the ESRP test results for this solution. Note: Detailed test result reports are attached to this PDF in HTML format.
We validated this solution using Microsoft Jetstress 2013 to ensure that the storage design satisfies the disk I/O and capacity requirements for the target profile. Jetstress simulates Exchange Server 2013 I/O types at the database level by interacting directly with the database Extensible Storage Engine (ESE) technology (also known as Jet) on which Exchange is built. Jetstress can be configured to test the maximum I/O throughput available to the disk subsystem within the performance constraints of Exchange. Alternatively, Jetstress can be configured to accept a specific user profile (user count, IOPS per user) and validate that the disk subsystem is capable of maintaining an acceptable level of performance with such a profile. Both configuration options produce similar results. Note: The Jetstress tool is designed to test performance of an Exchange storage subsystem before the subsystem is placed in a production environment. Jetstress is not designed to test server CPU, memory configuration, or the impact of user client activity. To test Exchange server functionality and end-to-end deployment, it is a best practice to use Microsoft Load Generator (LoadGen).
The documentation for Jetstress describes how to configure and execute I/O validation or evaluation on your server hardware. The Microsoft Jetstress tool is available at www.microsoft.com/en-us/download/details.aspx?id=36849. Storage reliability (stress test)
Storage performance
The reliability (stress) test runs for 24 hours. The goal of this test is to validate that the storage can handle high I/O load for a long period of time. Both log and database files are analyzed for integrity after the stress tests to ensure that no database or log corruption occurs. The 24-hour stress test results reveal that: •
No errors were reported in the saved event log file.
•
No errors were reported for the database and log checksum processes.
The purpose of the primary storage performance test is to exercise the storage with the maximum sustainable Exchange I/O for 2 hours with 100 percent user concurrency. The test reveals the amount of time it takes the storage to respond to I/O requests under load. This section presents the sum of all database transactional I/O and the average I/O latencies for all databases and logs recorded during the 2-hour test duration. Data for each server is presented separately. Aggregates across servers are also included. Individual server metrics Table 9 presents the sum of the I/O across all databases and the average latencies across all databases and logs for each server. The configuration is designed to achieve a target of 600 IOPS for 5,000 users on each server. This includes 20 percent overhead, above the 0.10 IOPS user profile, for reserve throughput capability. EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
22
Table 9.
Individual server metrics
Database I/O
EXCH01
EXCH02
EXCH03
EXCH04
Target transactional I/O
600
600
600
600
Achieved transactional I/O
685
682
685
680
I/O database reads/sec
472
469
472
468
I/O database writes/sec
213
213
213
212
I/O database average read latency (ms)
15.69
15.88
16
15.58
I/O database average write latency (ms)
1.72
2.07
1.81
1.76
EXCH01
EXCH02
EXCH03
EXCH04
54
54
54
55
0.59
0.61
0.55
0.58
Transaction log I/O I/O log writes/sec I/O log average write latency (ms)
Performance across servers Table 10 presents the sum of the I/O across servers and the average latency across servers. The configuration is designed to achieve a target of 2,400 Exchange Server 2013 IOPS for 20,000 users. The results show excellent I/O performance with 2,732 Exchange Server 2013 user IOPS achieved. This provides an additional 332 IOPS reserved throughput capacity to absorb any unexpected spikes during extremely heavy user activities. Note that these results already include 20 percent IOPS overhead recommended by Microsoft. Table 10.
Database backup and recovery performance
Performance across servers
Database I/O
Value
Target transactional IOPS
2,400
Achieved transactional IOPS
2,732
I/O database reads/sec
1,881
I/O database writes/sec
851
I/O database average read latency (ms)
15.79
I/O database average write latency (ms)
1.84
Transaction log I/O
Value
I/O log writes/sec
217
I/O log average write latency (ms)
0.58
Two tests are reported in this category. The first test measures the sequential read rate of database files, and the second test measures recovery/replay (the playing of transaction logs against databases) performance.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
23
Database read-only performance The database read-only performance test measures the maximum rate at which databases can be backed up through the use of VSS. Table 11 shows the average rate of backing up a single database file across four servers. Table 11.
Database read-only performance
Metric
Value
MBs read/sec per database
35.18
MBs read/sec per server
351.8
Total MBs read/sec for all four servers
1,407.2
Transaction log recovery/replay performance The transaction log recovery/replay performance test measures the maximum rate at which log files can be played against databases. Table 12 shows the average rate for 500 log files played against a single database. Each log file is 1 MB in size. Table 12.
Transaction log recovery/replay performance
Metric
Value
Average time to play one log file (in seconds)
2.78
Detailed test results Detailed test results for all four mailbox servers are attached to this PDF in the form of Jetstress HTML reports. Report types are as follows:
How to view Jetstress reports
•
24-hour stress test
•
Checksum for 24-hour stress test
•
2-hour performance test
•
Checksum for 2-hour performance test
•
Backup test
•
Soft recovery test
Click the paper clip icon in the left-hand pane of Adobe Reader to reveal the list of Jetstress reports (HTML files). Double-click each file to open the corresponding report in your browser.
EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
24
Conclusion This document is developed by EMC and reviewed by the Microsoft Exchange Product team. The storage validation performed and the data presented in this document are based on the tests introduced in the Microsoft ESRP test framework. The results were reviewed by the Microsoft Exchange Product team. Customers should not quote the data directly for their predeployment verification. It is still necessary to go through the exercises to validate the storage design for each specific customer environment. The ESRP program is not designed to be a benchmarking program; tests are not designed to achieve the maximum throughput for a given solution. Rather, the program is focused on producing recommendations from vendors for Exchange applications. The data presented in this document should not be used for making direct comparisons between various solutions. The testing and validation of this Exchange Server 2013 mailbox resiliency storage solution demonstrates the following: •
The EMC VNX5400 storage array is an excellent platform for Exchange Server 2013 storage. The array provides optimal performance with adequate capacity for deploying large Exchange mailboxes.
•
The VNX5400 array is fully optimized for Exchange Server 2013 workloads.
•
A building-block approach simplifies mailbox server design and facilitates scalable, predictable performance for all mailbox servers.
EMC recommends that you use a building-block approach when designing storage solutions for Exchange Server 2013. This solution uses a building block of 5,000 users with 0.12 IOPS per user (including 20 percent overhead) and a 2 GB mailbox size for validation. The configuration is scaled to 20,000 users through the utilization of four building blocks on an EMC VNX5400 storage array. This configuration meets all of the recommended Microsoft Exchange Server 2013 metrics for performance and capacity. In addition, the performance (I/O and latency) test results presented in this Microsoft-approved ESRP document demonstrate that this solution provides significant headroom to accommodate future user growth. EMC periodically publishes Proven Solutions white papers that demonstrate that the EMC VNX series of storage arrays can handle extremely heavy Exchange Server 2013 workloads. For more information, visit www.emc.com/exchange. Contact EMC
EMC recommends that you consult with EMC Professional Services to assist with the design and deployment of a similar solution. For information regarding this or any other EMC solution, use the following numbers: •
United States: (800) 782-4362 (SVC-4EMC)
•
Canada: (800) 543-4782 (543-4SVC)
•
Worldwide: (508) 497-7901
For additional information on EMC products and services available to customers and partners, visit www.EMC.com or http://Powerlink.EMC.com. EMC VNX5400 Unified Storage 20,000 Users Virtualized Microsoft Exchange Server 2013 Mailbox Resiliency Storage Solution
25
