Always Available Surveillance Video

Transcript

Always Available Surveillance Video W HITEPA PER www.pivot3.com Whitepaper – Always Available Surveillance Video Table of Contents Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The Simple Needs of Video Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 IT SANs are Built to Deliver an Average Level of Performance, Not a Minimum Level of Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 There are Three Ways You Can Lose Video Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Protecting Video is a Unique Challenge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Video is Different. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Some Practical Methodologies for Protecting Video . . . . . . . . . . . . . . . . . . . . . . . . 6 Hard Drives Fail Far More Often than Manufacturers Admit. . . . . . . . . . . . . . . . . . 7 Hyperconverged Appliances: The Key to Always Available Video . . . . . . . . . . . 9 Ensuring Continuity of Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1 Whitepaper – Always Available Surveillance Video Executive Summary The incident may have happened in the blink of an eye, or repeatedly over the course of days. In either case, did your video surveillance system catch it? What if a hard drive or even an entire server crashed? Can you still retrieve the evidence? Is the surveillance system still up and running while you replace the broken hardware? The real world of video surveillance requires: • Clear, 100%-available, live cameras. • Absolute certainty that what your cameras see is recorded. • 24X7 access to every frame of recorded video. Hyperconverged appliances enable you to pool computing, networking, memory, and SAN storage resources across several physical machines. As your surveillance needs grow, you can incrementally grow your computing and storage infrastructure one 3.4-inch (2U) appliance at a time. The first in this series of white papers outlines the Best Practices for Video Storage Infrastructure. It describes the unique considerations you have to take into account when developing video surveillance specifications. Concentrating on three storage technologies, it walks through the pros and cons of each specifically with regard to video data. It concludes that small sites are best suited for direct attached storage (DAS) systems and medium to large sites demand the much higher reliability, scalability, and data protection available from video-optimized storage area networks (SANs). This second white paper pulls the camera’s focus back a bit: after deciding to specify a SAN for your video surveillance system, how do you ensure operational continuity? Hard drives and other components fail far more often than their manufacturers would like to admit. What features should you specify to ensure your video surveillance system can handle a pixel storm amidst the silent, sudden death of a disc or an entire server? What other benefits should you expect (and require)? Is buying, installing, and configuring a SAN really as complex and expensive as it first appears? New technologies such as hyperconverged infrastructure are available to create a virtual SAN across three or more appliance servers. Could this approach offer a practical alternative to investing in a 1,500-pound IT SAN and a high-priced administrator to run it? Before you ask your IT colleagues what they would recommend, read the section on how Video is different, especially when it comes to protecting it. (Some strategies may seem initially counterintuitive to IT experts.) Video management software and hyperconverged appliances offer quite complementary failover capabilities. As you develop your specifications, you may find that you’ll want the features of both. Intrigued? Read on! 2 Whitepaper – Always Available Surveillance Video The Simple Needs of Video Surveillance It really is simple, isn’t it? You’re responsible for 24/7 live coverage of a certain geography, perhaps an entire building, campus, or site. Beyond that, the information your cameras are collecting needs to be recorded and stored so that you can immediately retrieve it when something happens. The problem, obviously, is making sure that your system infrastructure can always meet those needs. Murphy’s Law dictates that a hard drive or even an entire server will start hiccuping the moment a suspicious incident occurs. Require an expensive, complex infrastructure, right? The previous white paper* in this series outlined the development of different video storage technologies: direct attached storage (DAS), network attached storage (NAS), and storage area network (SAN). Digital video recorders and network video recorders with DAS proliferated because they were simple digital replacements for VCRs. DAS is terrific for small applications. However, these single-box machines with fixed storage capacity and performance cannot offer the scalability and reliability IT and surveillance professionals demand. NAS was designed to provide a much larger, more flexible storage resource for the readintensive needs of large businesses (email systems, Microsoft Office files, etc.). NAS offers better scalability than DAS, but it’s not a good fit for the write-intensive world of video. Its file system layer creates additional traffic on the same network you want to optimize for video data recording. In addition, file fragmentation (and NAS’s unending need to run a de-fragment program) severely affects its recording performance. * Best Practices for Video Storage Infrastructure, 2015. By contrast, SANs offer seamless consolidation and sharing of storage space, making them much more efficient than NAS devices. A SAN enables video data to go straight from the camera through the network to a generalized pool of storage. 3 Whitepaper – Always Available Surveillance Video IT SANs are Built to Deliver an Average Level of Performance, Not a Minimum Level of Service IT SANs are sold based on an average level of performance. However, video systems require a certain minimum level of performance at all times. As explained in more detail below, any slowdown in the system’s computing or storage writing ability can cause data loss. A video system cannot catch up from a momentary hiccup by going twice as fast sometime later. SANs are engineered to be highly reliable and scalable. However, their inherent design (perfect for a general-purpose IT environment with lots of small, random reads and writes) requires you to greatly over-provision them in order to manage the wildly-fluctuating barrage of incoming video from hundreds or thousands of cameras. In addition, IT SANs are very complex to set up and run, requiring the services of a specially-trained administrator. Reference frames (I-frames): periodic snapshots of an entire scene. Follow-up frames (P-frames): include only the pixels that changed since the previous I-frame If image quality is paramount, why not specify a system that can handle everything your expensive cameras can throw at it? There are Three Ways You Can Lose Video Data 1. Your system’s server crashes, stopping the live feed If your server goes down, you’ll immediately lose access to your live video feed. Not only do you become blind, you’ve completely lost the ability to record. While each camera might have onboard storage, its capacity is very limited, and there’s no way the system could “catch up” after you reboot the server (by recording both the stored and live video). 2. Your server cannot be repaired, stranding the video already recorded If you’re unable to restart the server that just went down, the video it holds immediately becomes inaccessible. You could hire a data forensics expert to try to recover the data, but that effort could take days or weeks. 3. Individual frames are lost because the system cannot record fast enough It’s impossible to read the license plate of the moving vehicle because the system cannot record the data fast enough. The reference frame (capturing unchanged pixels) was recorded, but not the follow-up frames. * A good analogy is what happens to satellite TV reception during a thunderstorm: a portion of the image pixelates, getting worse and worse as you lose both reference and follow-up frames. Each of your cameras compresses the raw scene that it sees before sending it to be recorded. (Two common compression formats these days are MPEG-4 and H.264.) These video compression methods have two hidden downsides: • If a single reference frame is lost, the follow-up frames that capture only the pixels that changed (not the entire scene) suddenly become gibberish. There’s a very real possibility that just one lost reference frame (I-frame) could mean the loss of several seconds of video. • When a pixel storm erupts from activity in a scene, the rush of data can overwhelm your system’s ability to record it. When this happens, you can also lose the follow-up frames (known as P-frames). The result is an image that’s pixelated, clearly showing objects that didn’t move, while the person or vehicle moving through the scene is blurred.* 4 Whitepaper – Always Available Surveillance Video Protecting Video is a Unique Challenge Lose one reference frame and you can lose seconds of video. Lose the follow-up frames and all the motion in a scene can be lost. When specifying a large, building-wide or campus-wide video surveillance system, obviously a lot of complexity is involved. If you have access to the resources of an IT department, it’s natural to tap their expertise in building big networks and protecting data. IT experts will tell you to be sure to specify a video surveillance system that performs: • data backups • replication • deduplication • compression • migration One question IT might suggest asking vendors is: what’s the average performance of the server measured in IOPS? All of these requirements and questions are perfectly suited to a system that handles IT data. They have nothing to do with video surveillance. Why? Because, as mentioned in Best Practices for Video Storage Infrastructure, video is different. Video is Different Surveillance cameras never stop sending huge, fat, streams of video data. Roughly 95 percent of the network traffic is writing to a disc. More than anything else, these two facts define why you have to think about protecting video in a completely different way than you would IT data. Backups Video is always coming into a surveillance system faster than it can be backed up. That simple fact negates a common IT approach to preserving data. Instead, the first consideration for safeguarding your video data should be to specify strict, minimal system performance requirements. In other words, when your cameras see a lot of activity, the system should be able to handle the pixel storm. Video also requires a high degree of system resiliency: if a hard drive, server, or application crashes, the system should automatically* recover from the failure. Replication The most common replication approach is controller-to-controller, which is found within traditional IT storage technologies from companies like EMC, Hitachi, and IBM. After the system writes data to a storage device, it then writes it to another location. A lot of IT departments keep an on-premise copy of the data and then a remote one for disaster recovery. * Within three minutes Less than 20 percent of an IT system’s traffic is new content, so the CPU performance and storage bandwidth requirements for replication are quite low. By contrast, replicating video 5 Whitepaper – Always Available Surveillance Video data would be a virtually impossible task, given the fact that huge volumes of new, sequential, data are constantly being written to disc. Compression and deduplication Video cameras themselves, together with your video management system (VMS), define how the video data is compressed (See Individual frames are lost because the system cannot record fast enough, on page 4). Applying additional compression to this already-compressed data generally has little effect. Deduplication also doesn’t make any sense when it comes to video surveillance data. (There aren’t any duplicate copies of streaming data.) After the retention period, the expired video is overwritten with new content. Migration If your building or campus has an older surveillance system, the best practice regarding your existing video is to simply allow it to age until the retention period ends. This approach saves a lot of time and money, while ensuring your old video data is available if necessary. Some Practical Methodologies for Protecting Video Realizing the severe shortcomings of IT best practices, the video surveillance industry responded by creating a handful of tools and strategies for improving system resilience and data protection. VMS failover VMS software developers build robust (often optional) failover features into their applications. For example, the VMS instance on one server is reading all the data from your site’s cameras. Elsewhere, a directory server is monitoring the system’s health. If it detects that the VMS is not reading from the cameras, it’ll quickly* assign the cameras a different VMS instance (on another machine). The main advantage to this feature is that it doesn’t matter why the system went down, whether it was a bug in a software application, Windows crashed, or the box on which it was loaded died. The directory server simply assigns another VMS instance to read from the cameras. One disadvantage is that when a failover occurs, the video on the broken-down hardware becomes inaccessible (stranded**). In addition, VMS failover features can be an expensive option. * Depending on the VMS vendor, the software failover can take anywhere from 15 seconds to a couple of minutes. ** Another downside: you have to leave spare capacity on your other servers to enable the cameras to be moved. Dual-streaming If you absolutely have to have a copy of everything, specify that the cameras dual-stream content to two SANs at different sites. In cases where overall bandwidth is limited, video can be streamed at a high resolution to the local site, and a low resolution to the backup site. 6 Whitepaper – Always Available Surveillance Video For example, if you wanted to cover two locations, say a downtown office and an airport warehouse, you could have all the office-based cameras send a second stream of data to warehouse (and vice versa). If the office datacenter went down, you could still monitor and record both sites from the warehouse without interruption.* Microsoft Cluster Service (MSCS) MSCS1 is software that enables computer servers to work together as a cluster. The technology provides network and component load balancing, providing automatic failover if a server crashes. While this approach is often considered for video, it’s rarely used because of its complexity, fragility, and expense. All-in-all, most of these failover technologies are expensive, often requiring additional licenses, significantly more bandwidth, and spare hardware (doubling your server and storage investment). Nevertheless, these various technologies weren’t developed to handle some rare event. Fortunately, a newer, much more flexible technology is available that offers the pooledresource benefits of a SAN without requiring the huge capital investment and operating expenses of a six-foot tall, 1,500-pound machine. Hard Drives Fail Far More Often than Manufacturers Admit You cannot escape the annualized failure rate (AFR) of hard drives: 2 to 13 percent! If you’re wondering just how likely it is your video surveillance system will experience hard drive failure in a given year, there are two sources of data: the manufacturers themselves and independent studies. Digging into the hard drive manuals, you’ll find that, on average, manufacturers claim an annualized failure rate (AFR) of about ½ a percent2. In other words, if you had 200 drives, you can expect just one to fail in a year. However, two academic studies and several large-scale cloud service providers3 have found otherwise. Beginning in 2001, a team at Carnegie Mellon University studied replacement data for 100,000 hard drives in large production systems over a period of five years. They found AFRs commonly were between 2-4 percent. Some systems experienced AFRs as high as 13 percent.4 Interestingly, failure rates seemed to have little to do with the type of drive (SCSI, Fibre Channel, or SATA). And, while the replacement rates declined after the first year of service (after an initial wave of “infant mortality”), they didn’t stay flat** for long.5 ** At two to four percent. In the mid-2000s, a team from Google studied more than 100,000 hard drives in the company’s data centers.*** While manufacturers claim that three out of ten returned drives have “nothing wrong with them,” customers often have different criteria for failing a drive.6 In the case of video data, as soon as a drive begins to falter, it becomes a serious problem. Why? Because the content never stops coming. *** Google’s team found AFRs from 1.7 percent for one year-old drives to 8.6 percent for three year-olds. Compounding the problem is the way that a drive fails. It seldom winks out like a light bulb: one moment it’s working 100 percent, then the next moment it’s not. Instead, the drive * In addition to being a lower resolution, the second data stream (from the office to the warehouse, in this example) is typically not retained as long as it is at the primary site. 7 Whitepaper – Always Available Surveillance Video typically starts at 100 percent, then begins degrading, getting slower and slower. In a normal IT datacenter environment, when the system becomes overwhelmed, users experience slow response times. If you slow down even a little bit in the video world, that could turn into a data loss. If it wasn’t fast enough to catch all that data as it came off the camera, it’s gone forever.* While hard drives are generally reliable, they nevertheless are subject to failure just like any component hardware. (In fact, they topped the list of ten most frequently-replaced components in the Carnegie Mellon study, followed by memory, CPU, and motherboard.)7 To be sure your system is capturing what your cameras see, you have to spec a system that’s not only able to handle the inevitable pixel storm, but can gracefully retire failed drives or even an entire box. IT SANs have redundant controllers and offer RAID protection for their hard drives (which make up the bulk of the large cabinet’s guts). However, they typically don’t support other types of component failures. Fortunately, there’s an alternative to buying two 1,500-pound SANs, hiring a high-paid SAN administrator, and taking weeks to optimize the machines for video. * Completely absent, or pixelated and blurred because important frames are missing. 8 Whitepaper – Always Available Surveillance Video Hyperconverged Appliances: The Key to Always Available Video A video surveillance system is a lot like the basic utilities (electric, gas, and Internet service, for example) that enable your site to run: it has to be on and performing at its minimum level of service all the time. To be really useful, it also needs to be fairly simple to operate and maintain. Well-designed surveillance systems always have a lot of underlying complexity to ensure they are always available, resilient, and scalable. But, that complexity has to be hidden (automated) so that your people can focus on security. Rather than hiring a high-priced SAN administrator to optimize a huge, expensive, complex IT SAN for video, consider instead specifying a SAN that was designed from the ground-up for surveillance video: one that employs hyperconverged appliances. A video-optimized SAN, designed for mere mortals As explained in Best Practices for Video Storage Infrastructure, virtualization software allows you to host several virtual CPUs on a single box. This technology is core to designing a system that can automatically failover when a component begins stuttering. Hyperconverged appliances are designed with a simple, real-world fact in mind: hard drives and other components fail. Hyperconvergence takes virtualization a step further. It abstracts the CPU and the other computer infrastructure components within a single machine: RAM, storage, and network cards, for example.8 However, not all hyperconverged solutions are created equally. When evaluating solutions, be sure that your hyperconverged platform leverages erasure coding rather than data replication for fault tolerance, and is built with a Distributed Scale-Out Architecture, which enables you to virtualize computing, networking, memory, and storage resources across more than one physical machine, creating a SAN with a brain – to differentiate between standard hyperconverged offerings, we’ll call this dynamic hyperconvergence. As your video surveillance needs grow, you can incrementally grow your generalized pools of CPUs, RAM, and video storage one 3.4-inch tall (2U) server appliance at a time. 9 Whitepaper – Always Available Surveillance Video Ensuring Continuity of Operations Without question, you want the most fault-tolerant video surveillance system you can get for your budget. As we’ve seen, hard drives (and even entire systems) are pretty reliable as complex components go. However, Murphy’s Law has a habit of invoking itself just when you need things to work perfectly. With dynamic hyperconvergence, all of the resources of your server appliances are pooled. When there’s a major incident and you need additional computing power, it’s there. If a physical server chooses that moment to crash, dynamic hyperconvergence treats it simply as a RAID event (like a single hard drive failed), thereby ensuring your operations continue with no adverse effect to your stored video data. APPLIANCE 1 APPLIANCE 2 APPLIANCE N CPU CPU CPU VM + VMS VM + VMS VM + VMS Aggregated HDD – Virtual SAN Aggregated SSD & RAM Cache – Write Optimized Distributed Data Protection – Erasure Coding Built–In Server Failover Scale-Out Server and Storage By specifying a high level of fault tolerance in your system’s hardware, you can ensure that it’s always capturing video and the data is always available to your surveillance team. An automated system that constantly monitors every disc, ready to shift work away from one that’s beginning to falter, ensures that new, incoming data isn’t lost. Older data on the failing disc is automatically copied to healthy ones through an advanced predictive sparing technology made possible by hyperconverged appliances. An advanced predictive sparing technology creates a virtual spare drive across all the hyperconverged appliances. Scalar erasure coding streams the data to the single pool of storage. Storage redundancy: data protection of the highest order Dynamic hyperconvergence enables you to protect your video data in a bulletproof, highly cost-effective way. An innovative technology called scalar erasure coding enables you to stream video a single pool of storage, across every disc in all the appliances in your video surveillance system. Whereas an IT SAN’s RAID system can protect you from up to two hard drive failures, erasure coding provides protection against five simultaneous disc failures, or the loss of an entire appliance and two additional drives in another appliance.9 Not only is your video data safe, it’s immediately available for viewing, too. 10 Whitepaper – Always Available Surveillance Video Pooling hard drives across hyperconverged appliances is key to high performance Recall that if your video surveillance system slows down, even for an instant, that can be a data loss event. This fact is why ingest performance is important: unless your systems is optimized to handle the flood of incoming data, you’ll lose it. When you pool hard drives across an array of appliances, you’ve created a system that’s perfect for handling a never-ending stream of video data. While IT SANs can only spread writes over the discs within the RAID group on a single machine, scalar erasure coding can write data across all the discs* in the array. This is a key feature to specify to reach the minimal performance your system requires to avoid dropping frames or entire video streams. The green line at left shows how full the disc queue** becomes in a hyperconverged appliance. In this real-world example, it’s averaging about 30 percent.*** The red line shows the CPU idle time****, indicating that overall application performance is very high. Computer redundancy: pooling all your servers’ resources Dynamic hyperconvergence also provides an important, often overlooked benefit of fault tolerance: computing power redundancy. Pooling overall performance, memory, and CPUs offers much faster data intake and processing. If you lose an appliance, your video surveillance operations continue without a hitch. This pool of computer power also provides enough spare capacity to easily run (and protect) other critical security systems, such as access control and video analytics. Hardware failover complements VMS failover Hardware and software vendors may debate whether the other’s failover features are really necessary. In fact, they complement one another. Each offers pros and cons that are well worth careful consideration. Depending on your site’s requirements, you may want one or both. PROS * Up to 144 standard (7,200rpm) SATA disks. ** The amount of data waiting to be written to disc. If the queue hits 100 percent, incoming video data can be lost. Hardware Failover VMS Failover • Handles an entire server going offline. • Preserves the integrity of your live system (cameras continue recording) while ensuring all your recorded data is continually available. • Hyperconverged appliance resources (CPUs, RAM, controllers, and storage) are pooled together and available at all times, not just when there’s a failure. • Hardware failover features are included in the base price of the system. • Handles an entire server going offline. • Can occur very quickly, meaning your recording may be interrupted for just 15 to 30 seconds. • The failover feature immediately assigns a new VMS instance to the cameras (no matter why there was a failure). *** When a disc queue reaches 100 percent, the system can begin losing data. **** An idle time of 100 percent is best. 11 Whitepaper – Always Available Surveillance Video CONS Hardware Failover VMS Failover • Requires 2 to 3 minutes, interrupting your recording during that time. • Only works for hardware failures. For example, if the VMS application crashes, the hardware failover processes are not alerted and cannot respond to the problem. • If something was wrong with the hardware, the data on that box becomes “stranded” until the machine is repaired (if possible). • VMS failover features are typically optional (not included in the base price of the software). • Requires spare hardware. If there’s a problem with the entire appliance, dynamic hyperconvergence can shift work from one hyperconverged appliance to another. Typically, the process of shutting down one VMS instance and starting it on another machine requires two to three minutes. (Most of that time, the system is waiting for Windows to start.) Again, all of the data that was on the broken appliance is available. However, your new recordings are interrupted for two to three minutes. If you can’t live with a two- to three-minute gap in coverage, and you want to be sure that software application failures are covered, you should definitely look into adding the VMS failover option to your surveillance system. If you also want continued access to the recorded data on the server that just failed, you should strongly consider using both VMS failover and hardware failover features. A scale-out design eliminates data bottlenecks Finally, dynamically hyperconverged appliances provide a unique architecture where system performance and storage bandwidth expand as you add cameras. The scale-out design distributes incoming video across all your appliances to dissipate pixel storms, eliminates single points of failure (including an entire appliance), and maximizes frame-rate capture for the best recording results. The scale-out design also simplifies management and speeds responsiveness to changing field requirements. 12 Whitepaper – Always Available Surveillance Video Recommendations While it’s certainly possible to configure an IT SAN for video surveillance, hyperconverged appliances offer a much simpler, more practical, and cost-effective alternative. By combining all of the computing and storage resources of small appliance servers, you gain worse-case operational continuity, incremental scalability, and very high level of data protection. Simply put: hardware breaks down, and software goes awry. You need a system that’s resilient, designed with those realities in mind. To ensure you spec the kind of video system that you and your surveillance team can depend on: → Begin with the physical requirements: what rooms, buildings, perimeters, and campuses do you need to cover? → How critical is capturing every minute of video to your mission? What about access to the recorded video on a server that failed? If you need 100 percent of both, consider specifying VMS failover and global convergence failover. → Are you responsible for two different locations? If so, you might want to consider dual-streaming, using each site as a second repository for the other. However, this approach will significantly increase your hardware and software investment. → Dynamic hyperconvergence offers both computing and storage redundancy. To protect the continued operation of your broader security system consider running some of your other applications (such as access control and/or video analytics) on the spare capacity the hyperconverged appliances offer. 13 Whitepaper – Always Available Surveillance Video Sources Introducing Microsoft Cluster Service (MSCS) in the Windows Server 2003 Family.” https://msdn.microsoft.com/en-us/library/ms952401.aspx, accessed July 3, 2015. 1 For example, Seagate says its 3-1/2-inch enterprise hard drive has an AFR of 0.44 percent. See “Seagate Enterprise Capacity 3.5 HDD v4 Serial ATA,” page 21. http://www. seagate.com/www-content/product-content/enterprise-hdd-fam/enterprise-capacity3-5-hdd/constellation-es-4/en-us/docs/100740544d.pdf, accessed on July 3, 2015. 2 For example, BackBlaze found a five percent AFR in its one year-old discs and a 12 percent failure rate in drives that were three years old. See Chacos, Brad. “25,000-drive study shines a light on how long hard drives actually last, PCWorld, Nov. 12, 2013. http://www.pcworld.com/article/2062254/25-000-drive-study-shines-a-light-on-howlong-hard-drives-actually-last.html, accessed July 3, 2015. 3 Schroeder, Bianca and Garth A. Gibson. “Disk failures in the real world:
What does an MTTF of 1,000,000 hours mean to you?,” page 1. https://www.cs.cmu.edu/~bianca/ fast07.pdf, accessed July 3, 2015. 4 Ibid., pp. 1, 14. 5 Ibid., p. 2. See also Pinheiro, Eduardo, Wolf-Dietrich Weber and Luiz Andre ́ Barroso. “Failure Trends in a Large Disk Drive Population,” p. 13. http://static.googleusercontent. com/media/research.google.com/en/us/archive/ disk_failures.pdf, accessed July 3, 2015. 6 Schroeder and Gibson, p. 6. 7 “Virtualization.” Wikipedia. http://en.wikipedia.org/wiki/Virtualization#Desktop_ virtualization, accessed Dec. 11, 2014. 8 The system described here can achieve up to six nines of availability (99.9999 percent). 9 © 2016 Pivot3, Inc. This document is for informational purposes only. Pivot3 reserves the right to make changes without further notice to any products herein. The content provided is as is and without express or implied warranties of any kind. 14