Transcript
Creating a shared workflow infrastructure for multi-suite, high resolution post production White Paper Overview The need for teamworking in high quality post and TV Finishing continues to grow as deadlines get shorter and budgets continue to shrink. Post production houses and broadcasters need to be able to utilize all of their resources to the fullest possible extent to create the maximum amount of client-facing (and so billable) time from each and every system and person.
What is Rio Connect? Rio Connect is a shared storage and workflow system that allows a number of SAM Rio color correction and finishing systems or Rio Assist systems to be teamworked; third party systems can also be integrated into the workflow. Rio Connect is built on COTS technology to keep costs down and take advantage of powerful GPU processing, flexible multicore CPUs and high performance storage.
The advantages a shared infrastructure can bring to the post facility are enormous:
A Rio Connect system comprises between one and eight SAM Rio or Rio Assist systems; the Rio Connect and the shared storage itself, which offers capacities in excess of 400TB of online storage. In addition, if third party systems are to be connected to the system, then a Rio Gateway replaces one of the SAM Rio clients.
•
Users have complete flexibility to move jobs between suites at a moment's notice
•
Users can teamwork across multiple suites instantly
•
Users can instantly lay off time-consuming jobs to backroom systems such as Rio Assist to free up time in their main suites for billable work.
•
Know that whatever is happening in another suite their system will deliver realtime performance
SAM has created Rio Connect in response to this need. This White Paper describes how SAM has employed COTS technology in developing Rio Connect to create a shared workflow infrastructure capable of sustaining up to eight edit suites without compromising realtime performance in any of them.
SAM Rio
SAM Rio
SAM Rio
Taking a step back for a moment to a single SAM Rio system, working with 4K 60p reliably in realtime is not trivial; Sony chose SAM Rio and the previous generation Genetic Engineering for the production of its 4K highlights packages at the FIFA World Cup in 2014 because: “The chosen system needed to handle 4K 60p in realtime, reliably and to produce the finished highlights packages quickly and efficiently…This made SAM a natural choice for this demanding application."
SAM Rio
SAM Rio
SAM Rio
SAM Rio
Customer 10G/1G Network Switch *
SAM Rio
Rio Connect
SAS Array Storage
SAS Array Storage
Rio Connect – Sample System
1
SAS Array Storage
SAS Array Storage
SAS Array Storage
SAS Array Storage
White Paper To deliver the required performance to support multiple SAM Rio systems operating in realtime at 2K, 4K and beyond with the same storage, SAM has developed a number of novel techniques which together guarantee performance for all connected systems. Guaranteed realtime performance is critical to post production if the full advantages of shared workspace and workflows are to be realized. The data rates required to support realtime operation with the ever increasing image sizes and higher frame rates that modern post production has to handle are considerable, and while 4K 60fps is now becoming common, 6K is beginning to make inroads following the introduction of the RED Dragon camera and even 8K with the RED Weapon camera; In Japan 8K is already in use and at 60 frames per second with a desire to move to 120fps! The table below shows the critical data transfer rates required to deliver realtime operation. Format
Image size (MB/Frame)
Transfer rate (GB/s)
Storage (TB/hour)
4k 10 bit 24 fps
33
0.8
2.9
4K 16 bit 24 fps
53
1.3
4.8
6K 16 bit 24 fps
116
2.8
10.1
4K 16 bit 60 fps
53
3.2
11.5
8K 16 Bit 60 fps
212
12
45.9
8K 16 bit 120 fps
212
24
91.7
High-end post format data requirements. 4K in the above table refers to 4096 x 2160. 16 bit to half-float or 16 bit integer
There are three key considerations for overall system performance to deliver the massive data rates required: disk performance on the shared storage, infrastructure performance, and the impact of the clients themselves. High performance storage Taking 4K 60p as the benchmark, a Rio Connect system with four SAM Rio clients needs to be able to reliably deliver 2000MB/s and will require 7.2TB of storage per hour of media. While this could in theory be achieved by using a commercially available SAN, SAM has not chosen this route for Rio Connect because SANs are designed to expose file systems to the clients. SAM Rio storage uses block addressing – it doesn’t require a file system. A SAN would simply add an additional layer that is not required, adding unnecessary complexity and creating a potential performance bottleneck. 2
SAM has chosen enterprise class SAS storage for the Rio Connect. Disks are 2.5” Seagate 10,000rpm in Xyratex enclosures, controlled by ATTO RAID controllers. Each EBOD contains 2x12 disk arrays with data striped across both arrays. Each array is configured with dual parity drives so the configuration is RAID 60. Commercial RAID controllers do not offer the same level of performance under fault conditions as when everything is working normally, for example when a drive has failed. This is factored into the system design so that sufficient performance is available even with one disk failed per array to sustain realtime operation across the system. Dual parity ensures that even under this fault condition the media is still protected. Should a second disk fail in any one array then the RAID controller performance reduces still further to a point where realtime performance cannot be maintained. However this is a critical state as the media is no longer protected so the faulty drives need to be replaced urgently. However even a modern RAID controller cannot deliver the data rates required by high end post so Rio Connect uses up to three RAID controllers per Rio Connect appliance to deliver the necessary bandwidth. SAM software processes the data in such a way that, for example, with three EBODs providing shared storage, seek times remain the same as for data stored on a single EBOD. This is a key differentiator for both SAM Rio and Rio Connect, and one of the reasons why SAM is able to extract more performance from off the shelf hardware than other companies. The array/RAID controller combination used in Rio Connect is able to sustain read and write data rates of 2GB/s per second under normal conditions. This is de-rated to 1400MB/s to allow the system to operate normally with a single drive failed in any array. The Rio Connect runs a linux application that allows multiple SAM Rio clients to share the same arrays without conflict. It also contains the audio array and a Redundant Hyperspace Service runs in a Windows VM. Its aggregate throughput is up to 4200MB/s hence each Rio Connect appliance can support three RAID controllers/arrays. More storage can be added by daisy chaining multiple arrays – this does not increase performance. To increase performance more RAID controller/array combinations must be added via an additional Rio Connect. In GE2 today the maximum is two Rio Connects, each with three RAID controllers giving a total bandwidth to storage of at least 8.4 GB/sec even under fault conditions. High performance infrastructure The clients are connected to the Rio Connect by 16Gb fiber channel. Fibre channel has proved itself over many years and continues to have a strong development roadmap. Single, Dual and Quad FC adaptors are readily available and FC provides a good match to the data rates required.
White Paper Rio Connect requires at least one FC connection to the Rio Connect appliance. Audio and video use the same fiber. A quad FC HBA in the Rio Connect can therefore support four clients. For high resolution and high frame rates the data rate required exceeds the capability of a single 16Gb fiber, which is 1600MB/s. A quick glance at the earlier table shows that 6K at 24 fps and 4K at 60 fps are well beyond 1600MB/s. However two fibers can support both these formats. So for systems requiring realtime performance with these high resolution formats, two fibers are used for image data – generally each fiber will connect to a different Rio Connect appliance. SAM Rio Clients SAM Rio systems are individually capable of viewing and playing out 4K 50/60p media in realtime; in fact, many customers are already working with 6K 50/60p on the system, also in realtime. They do this 100% reliably, with no stutters or drop outs. SAM Rio systems are also capable of realtime color correction on multiple cascades, and/or carrying out a variety of other production tasks also in realtime. They achieve this by taking advantage of the latest NVIDIA Maximus technology which enables multiple Tesla GPUs to work together in the same system. The current generation of Tesla GPUs used in SAM Rio is the K80, each with 4992 CUDA cores capable of delivering 8.74 teraFLOPS. In a shared environment the massive computing power in SAM Rio needs to be factored into the complete system design. SAM Rio can easily saturate two 16Gb fibers and pull 3200MB/s from the storage. Designing a system that can handle that demand and sustain realtime performance across all the clients is possible but would increase the storage and infrastructure cost. In a practical application Rio Connect systems are designed to support a specific maximum realtime format; for example for UHD 10 bit 422 60p it is 1400MB/s. A client saturating the FC connection would be demanding more than 2 times that data rate. Resolving this issue requires that the clients are ‘well-behaved’ and do not ask for more than their design limit bandwidth. As all the clients that directly connect to Rio Connect are SAM Rio clients they include a mechanism to limit the disk bandwidth they can demand. This limit is calculated from the system design and is part of the system configuration when it is installed.
3
The use of FC as the Rio Connect/client interconnect opens up the possibility of supporting more clients in Rio Connect systems in the future; however it is not as straightforward as it first seems. From a bandwidth perspective a FC switch does a great job of sharing that bandwidth amongst clients. However bandwidth is only one parameter in a shared storage system; as the number of clients increases the seek time becomes the limiting factor – more clients equals more seeks. This is an issue for tomorrow; today eight clients all doing UHD on shared storage with guaranteed realtime access is enough. Damon Hawkins SAM August 2015
Please contact
[email protected] with any questions.
