Transcript
31.08.2011 Test Report Quality Evaluation of the HDTV H.264/AVC Encoder CodecSys Dagmar Driesnack, Dr. Rainer Schäfer, IRT Final Report V1.0
___________________________________________________________________
The tests performed in the frame of this report were conducted on the order of Broadcast International Inc., IRT, offer no 281-10
Copyright Notice This document and all contents are protected by copyright law. All rights reserved. You may not reproduce, translate, distribute, microfilm, transmit, display, publish, alter or roll in and/or edit on electronic devices without prior written permission of Institut für Rundfunktechnik any parts of the document. You may not alter or remove any trademark, copyright or other notice from copies of the document. We grant you the right to copy and distribute to third parties (also electronically solely in noneditable .pdf format) as often as you like the complete and unchanged subject test report.
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
2 of 18
___________________________________________________________________
Content 1 Executive Summary............................................................................................... 5 1.1
Introduction to IRT ..........................................................................................................5
1.2
Test Summary ................................................................................................................ 5
2 Evaluation of Video Quality .................................................................................. 6 2.1
Description of evaluation method ................................................................................... 6
2.1.1
Basic Principle ...................................................................................................................... 6
2.1.2
Infrastructure ......................................................................................................................... 6
2.1.3
Subjective adjustment of PSNR figures ................................................................................ 8
2.1.4
Risks and inaccuracy .......................................................................................................... 10
2.2
Test sequences ............................................................................................................ 10
2.3
Encoder under test ....................................................................................................... 11
2.4
Encoding parameters ................................................................................................... 11
2.5
Results of quality evaluation using the HD test sequence ........................................... 12
2.6
Remarks on coding quality ........................................................................................... 12
2.7
Conclusions for real-time operation.............................................................................. 12
2.8
Additional test ...............................................................................................................12
3 Compatibility and Conformance Tests .............................................................. 13 3.1
PSI and DVB-SI Analysis ............................................................................................. 13
3.2
Signalling of aspect ratio through VUI parameters ....................................................... 13
3.3
Additional Remarks ...................................................................................................... 13
4 Annex A: Diagrams HD-Testsequence .............................................................. 14 5 Annex B: PSNR-comparison online-offline encoding ...................................... 16
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
3 of 18
___________________________________________________________________
[This page intentionally left blank]
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
4 of 18
___________________________________________________________________
1 Executive Summary 1.1
Introduction to IRT
The Institut für Rundfunktechnik (IRT) is the central research and development centre for 14 public broadcasting companies in Germany (ARD, ZDF, DRadio), Austria (ORF), and Switzerland (SRG/SSR). Since its foundation in 1956, IRT has been committed to preserving broadcasting and accompanying the adjustment of the broadcasting idea to new market environments and requirements. Further, IRT is cooperating with numerous clients from the broadcasting, media, communications, and information technology industries, as well as with various research institutions and academies. With its offices in Munich, IRT supports broadcasting on a national and international scale with its spectrum of services. With more than 100 engineers IRT covers all topics related to digital media technology – from TV and radio production systems via network technologies as well as internet applications to transmission systems and frequency management. The experts from IRT represent German broadcasters in major national and international standardisation bodies, such as the International Telecommunication Union (ITU), Digital Video Broadcasting Project (DVB), WorldDMB and the European Broadcasting Union (EBU). Examples where IRT has played a key role in system development and successful introduction into practical operation are: 1967 Electronic slow motion: Television scenes can be instantly replayed 1974 ARI traffic radio: Broadcasting of traffic information 1975 Teletext: Text information embedded in the television signal 1985 Eduard-Rhein-Award for VPS: Video programming system for video recorders 1988 Digital Radio: First DAB transmission 1992 HDTV via satellite: Digital HDTV broadcasting via satellite 1994 Development of first MPEG-2 encoder 1999 KEM omni-directional microphone: More freedom of movement for speakers and artists 2000 Emmy Award: Development of the ISO MPEG Layer II audio encoding standard 2004 HDTV: First uncompressed HDTV recordings in 720p/50 2006 Regional Radio Communications Conference: Realignment of the broadcast frequency spectrum for digital terrestrial radio communications 2009 Hybrid-TV: HbbTV-standard merges TV-programmes with the internet
1.2
Test Summary
The evaluation confirms a good coding quality of the of the CodecSys software-based encoder provided by Broadcast International when compared to a representative set of distribution encoders currently available on the market. When operating in real-time, it provides a quality near to the best hardware accelerated encoding systems. When operating in non-real-time, for progressive mode (720p/50) the quality is further optimized and is close to one of the best known state-of-the-art hardware accelerated distribution encoders. The encoder is one of the best software-based encoders tested by IRT up to now. The encoded bit-stream is compatible with professional receivers typical for the German market. With an adequate adjustment, it fulfils the pre-requisites for DVB broadcast (assuming that some DVB-specific Service Information will be added at a downstream multiplexing stage).
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
5 of 18
___________________________________________________________________
2 Evaluation of Video Quality 2.1
Description of evaluation method
2.1.1 Basic Principle There exists a series of methods for picture quality evaluation, amongst them the subjective tests according to ITU, methods proposed by the SAMVIQ group, or other methods like the JND metric implemented in products available in the market for SD quality evaluation (PQA 200 / Tektronix). It is well known that in general all automated methods do not correctly reflect the subjective assessment and that pure subjective assessment is extremely costly and timeconsuming. So-called “Expert views” are a method to reduce the resources required for subjective testing, but bear the risk that a result is not backed by enough statistical data. Therefore, the codec performance has been evaluated in a dual-step approach that combines - an automated PSNR measurement with - a correction of the PSNR figures by subjective assessment during an experts view. This method also allows to evaluate longer sequences than just 10s as in many other approaches for subjective testing. 2.1.2 Infrastructure Fig. 1 gives an overview on the infrastructure used for the codec evaluation: All test sequences were available as uncompressed YUV 4:2:2 (DPX, 10 bit) sequences and had never been compressed before. Each test sequence was preceded by a short so-called “align-sequence” that allows to automatically identify the first frame of the sequence for an automated downstream PSNR analysis, as well as to detect gain or contrast mismatches. For evaluation of hardware encoders, the signal was fed via HD-SDI links to the encoder under test and the encoded transport stream was recorded. All recorded transport streams were live decoded to DPX files including YUV 4:2:2, 10 bit by a hardware decoder and recorded on a transparent server (DVS Clipster). The decoded signal was then used for the PSNR calculation. It was confirmed that no other processing, such as colour space transformations, implicit compressions or wrapping to any other file format was applied.
SVT/EBU Sequences in 720p/50 and 1080i/25 (uncompressed dpx files)
Server (Clipster) uncompress ed
HDSDI
HardwareEncoder
ASI
Single Program Transport Stream
TS
TS-Server, Multiplexer -> two programmes for comparison
IRD
Display left
IRD
Display right Subjective comparisons, demos
IRD
dpx (uncompressed)
PSNR calculation
PSNR diagrams
Figure 1 Infrastructure codec quality evaluation Fig. 1 Infrastructure forfor codec quality evaluation
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
6 of 18
___________________________________________________________________
The PSNR values were calculated on a frame by frame basis. A white level of 1023 (corresponding to signal values from 210-1 in 10 bit systems) was applied as peak reference. For calculation of average PSNR figures for the whole sequence, the mean square error for all frames was added and then the logarithmic results of all frames were averaged according to the following formula:
frames 1 1 PSNR 20 lg frames 0 1 lines pixel
1023 lines 1 pixel 1
2
coded reference 0
0
For subjective adjustment and confirmation of the PSNR results, two selected streams from the set of recordings were multiplexed, modulated and decoded by professional decoders (Tandberg RX1290, software version v3.12.0) which were connected via HD-SDI links to two identical 50’’ PDP Flat Panels (Panasonic TH-50PF9EK). The displays were mounted next to each other in a viewing room with controlled lighting conditions (Figure 2). The “expert viewers” were able to select and play any of the recorded streams for direct comparison through a special user interface (Figure 3). The play-out of the two transport streams was synchronised to less than one second difference.
Figure 2 Setup for experts viewing
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
7 of 18
___________________________________________________________________
Figure 3 Example of the user interface for comparison of test streams
2.1.3 Subjective adjustment of PSNR figures The method applied to adjust results from PSNR calculation according to subjective assessment is described by the following steps. A sample result of the PSNR calculation from two different encoders is shown in Figure 4 as the blue and red curves. In case that the objective PSNR figures directly correspond to the subjective evaluation, sequences with the same PSNR, e.g. the “blue” (reference) encoder at bit-rate a) and the “red” encoder under test at bit-rate b) should give identical subjective quality when played and compared side by side on the two displays with a viewing distance of three times the picture height. In case that the subjective quality impression did not match, the comparison was repeated at different bit-rates until the subjective quality was estimated to be identical. In Figure 4, this is assumed for bit-rate c) for the “red” encoder.
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
8 of 18
___________________________________________________________________
35
PSNR (Y) [dB]
a)
c)
b) d)
30
25 5
10
15
20
Bitrate [M Bit/s]
Figure 4 Example for subjective adjustment of objective PSNR figures
As a consequence, a vertical correction of the “red” curve (d) towards lower PSNR figures can be identified and the objective PSNR diagram can be converted to a diagram representing subjective quality (Figure 5). The Y-axis of the new diagram does no longer represent pure absolute PSNR figures. However, a relative “visibility threshold” corresponding to 0.5-0.7 dB could be identified which matches very well figures known from literature. 5
a) b)
c)
0
5 5
10
15
20
Bitrate [MBit/s] Figure 5 Diagram after subjective adjustment
This subjective adjustment was carried out for each encoder under test against the same reference encoder Cisco D9050 (MPEG-2). Moreover, the adjustment offset was confirmed for each encoder at various bit-rates, e.g. to confirm that a pure vertical offset does really reflect the subjective assessment. As a final verification, direct cross-checks were applied between two “adjusted” encoders, e.g. between two different H.264 encoders in order to confirm that the
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
9 of 18
___________________________________________________________________
correcting offsets separately identified for these two encoders lead to consistent results when these two encoders are compared directly. 2.1.4 Risks and inaccuracy The applied method still and intentionally relies on subjective comparisons, and removes the need to translate subjective quality into distances on a certain quality scale. Dependently on the sensibility of viewers with respect to quantisation artefacts, blocking, blur, temporarily changing quality due to buffer control, different viewers might still come to slightly different conclusions. However, the overall rating has been jointly confirmed by 4-5 experts. Since the method provides very accurate figures also below the aforementioned visibility threshold of 0.5 to 0.7 dB which has also been confirmed in these tests, the results might pretend more granularity or differences between encoders, which would not be visible to test persons any more. This should be taken into account when interpreting the diagrams. This effect does not invalidate general conclusions such as bit-rate savings by choosing a certain encoder model.
2.2
Test sequences
The following test sequence was used for quality evaluation according to the described method. Mixture of different sequences (sports, show, documentation,… length about 6 minutes) - This material is used for different tests within ARD/ZDF and EBU. There are different genres included. The sequence reflects a good overview of HDTV. - This sequence is used to get an impression of the encoder performance in a practical environment. The test sequence is of substantial length and consists of several shots. Sample screen-shots are given in Figure 6.
Figure 6 Sample screenshots from HD test sequence
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
10 of 18
___________________________________________________________________
2.3
Encoder under test
Three different hardware encoders were used to compare the HD-quality of the Broadcast International CodecSys (version 2.6, rev 6146), amongst them the Cisco D9054 and the Ericsson EN8190 and the Cisco MPEG-2 encoder D9050 as the MPEG-2 reference. For reasons of neutrality, no direct reference to a specific product is given in the diagrams annexed to this report with the exception of the CodecSys and the MPEG-2 reference. However, it can be confirmed that both competitors are state-of-the-art encoders which were used by the broadcasters today.
2.4
Encoding parameters
All encoders used in the test were adjusted nearly in the same way: - any pre-processing (noise reduction, etc.) disabled, if available - GOP length for reference encoders was set to 16 for 1080i/25 and 32 for 720p/50, dynamic/ hierarchical GOP enabled if available. For the CodecSys system, by default a maximum GOP length of about 1s was configured. It was also confirmed for several reference streams, that the quality of the CodecSys encoded does not significantly degrade when fixed GOP lengths of 16 or 32 were configured. - High Profile - Full horizontal resolution - Constant bit-rate CBR
Figure 7 CodecSys Encoder configuration
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
11 of 18
___________________________________________________________________
2.5
Results of quality evaluation using the HD test sequence
Annex A gives the results of the objective PSNR quality measurements and the subjective adjustments applied. The Encoder CodecSys has been tested and compared to other representative encoders available on the market (see 2.3). For the objective PSNR figures, the CodecSys encoder shows an improvement over the whole range of bit-rates. Up to 8 Mbit/s the encoder shows the best objective quality in 720p/50 which is the relevant distribution format for the public German speaking broadcasters.
2.6
For the subjective evaluations, the encoder shows the second best picture quality in both formats. There is a bit-rate saving of about 50% compared to MPEG-2.
Remarks on coding quality
The encoder shows a well performing buffer control, i.e. there was no significant GOP pumping visible for a wide range of bit-rates. In addition, no disturbing effects e.g. on scene changes were observed.
2.7
Conclusions for real-time operation
The evaluation confirms a good coding quality when compared to a representative set of distribution encoders currently available on the market. When operating in real-time, it provides a quality near to the best hardware accelerated encoding systems. The encoder is one of the best SW-based encoders tested by IRT up to the preparation of this report.
2.8
Additional test
The CodecSys implementation is also able to encode HD source material offline from file. A short part of the whole test sequence was selected to compare this quality against the online encoding quality. Three bit-rates were tested (3, 6 and 10 Mbit/s) for both, 720p/50 and 1080i/25. The processing time required for optimum coding quality is about two times realtime. The subjective picture quality of the offline encoded progressive material is visibly better compared to the online encoded material, independent on the bit-rate, although this effect cannot be confirmed with PSNR-figures (see Annex B). Non real-time encoding improves the subjective picture quality especially for progressive 720p/50 sources by up to about 20-30%, whereas no significant improvement was found for 1080i/25.
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
12 of 18
___________________________________________________________________
3 Compatibility and Conformance Tests 3.1
PSI and DVB-SI Analysis
An analysis of the output transport stream was carried out with the streams recorded at the time when the encoder was available for testing. According to the DVB TR 101 290 v1.2.1 there are no errors for the first and the second priority. Only errors with the third priority occurred: - There are no NIT, SDT, EIT and TDT included in the transport streams. According to DVB (EN 300 468 v1.11.1) all these tables are mandatory. However, these tables are typically inserted in the main multiplexer. The stream_type 0x1B (27) for signalling of an H.264/AVC HD emission is correctly inserted.
3.2
Signalling of aspect ratio through VUI parameters
The H.264 standard (ISO IEC 14496-10) allows signalling of the aspect ratio by transmitting the pixel aspect ratio (aspect_ratio_idc) as an optional parameter within the so-called Video Usability Information (VUI). The DVB specification (ETSI TS 101 154 V1.10.1, 2011-06) requires the aspect_ratio_idc to be transmitted. It was confirmed that the CodecSys correctly inserts the VUI parameters including the aspect_ratio_idc parameter into the H.264 elementary stream for the supported resolution according to the following table. Format
720p/50
720p/50
1080i/25
1080i/25
1080i/25
1280x720
960x720
1920x1080
1440x1080
1280x1080
Aspect_ratio_info_present_ flag
1
1
1
1
1
Aspect_ratio_idc
1
14
1
14
15
Resolution
3.3
Additional Remarks
It was found, that the PCR distance parameter deviated slightly (increased) from the preconfigured value for higher bit-rates above 12 Mbit/s. This can be overcome by changing the configuration, or possibly by a future software update.
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
13 of 18
___________________________________________________________________ 4
Annex A: Diagrams HD-Testsequence
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
14 of 18
___________________________________________________________________
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
15 of 18
___________________________________________________________________
5 Annex B: PSNR-comparison online-offline encoding
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
16 of 18
___________________________________________________________________
© 2011 Institut für Rundfunktechnik GmbH. All rights reserved.
17 of 18
Institut für Rundfunktechnik GmbH Floriansmuehlstraße 60 80939 Munich Germany www.irt.de Phone +49 (0) 89 | 323 99 - 300 Fax +49 (0) 89 | 323 99 - 620
[email protected] Register Court Munich HRB 5191
