TMCnet News
Clarifying Video Quality MetricsThe successful deployment of IPTV (News - Alert) and videoconferencing services requires a range of tools for measuring and monitoring video quality. The impact of encoding and transmission impairments on the perceptual quality video streams is quite complex and depends heavily on the codec type and configuration, and on end system characteristics. There are a variety of algorithms for video quality estimation but relatively few standards and a fair amount of confusion.
There are essentially three “models” for video performance measurement.
Full Reference Algorithms
Full reference algorithms perform a detailed comparison of the input and output video stream. This is a computationally intensive process as it not only involves per-pixel processing but also time and spatial alignment of the input and output streams. Full reference algorithms can achieve good levels of correlation with subjective test data however can only be used in certain applications - for example in lab testing, pre-deployment test or troubleshooting.
One of the earliest full reference algorithms is PSNR (Peak Signal to Noise Ratio) [1], which is literally a measurement of the mean error between input and output as a ratio of the peak signal level, expressed in dB. A typical “good” PSNR is around 35dB and it is generally accepted that PSNR values of less than 20dB are unacceptable. PSNR is the most widely used technique for image and video quality measurement.
A wide range of full reference algorithms have been developed, including MPQM (Moving Pictures Quality Metric - 1996) full reference algorithm from EPFL in Switzerland [2], the US Government NTIA ITS lab’s VQM (Video Quality Metric - 1999) [3] and CVQE (Continuous Video Quality Evaluation - 2004) [4] which is more suited for low bit rate video. Of these three algorithms, the only one that is contained within a standard is VQM, which is incorporated into ITU-T J.144.
The Video Quality Experts Group (VQEG) has been actively working on objective video quality assessment since 1997, and generally acts in cooperation with ITU. VQEG has conducted two phases of testing; in the first phase ten algorithms were tested and the conclusion reached was that most of the algorithms (including PSNR) were statistically equivalent [5]. The second phase of testing, conducted several years later, involved a smaller number of algorithms and concluded that these did achieve good enough results to warrant recommendation for use, resulting in a recent ITU-T Recommendation J.144 [6].
ITU-T J.144 does not actually specify a single algorithm but “provides guidelines on the selection of appropriate” techniques. J.144 does contain descriptions and test results for four full reference algorithms, and also included PSNR as a reference. The VQM algorithm from the US Government’s NTIA ITS lab achieved slightly better performance than the other algorithms listed.
Commercially available full reference software includes Opticom’s PEVQ [7] and Psytechnics PVA [8] (and others), which are incorporated into systems marketed by many test equipment vendors.
Zero Reference Algorithms
Zero reference algorithms are generally more suitable for in-service monitoring of video services as they can analyze live streams. This type of algorithm can consider fewer factors than a full reference algorithm however can be deployed in a much wider variety of scenarios.
Media stream based algorithms, such as Telchemy’s VQmon/SA-VM [9] and Psytechnics’ PVI [10] analyze the IP stream and video transport protocols, building up an assessment of video quality and expressing this as a perceptual quality score.
Telchemy’s algorithm is differentiated in its ability to analyze the time distribution of lost and discarded packets and to model the impact of transient IP problems on perceptual quality, based on their widely adopted VQmon technology.
Psytechnics has an extensive history in subjective and objective testing of voice and video, dating back to roots in the British Telecom Research Labs. Both companies have extensively benchmarked their algorithms against both objective and subjective test data.
This type of algorithm can be very computationally efficient, taking a fraction of a MIP for processing, and is suitable for integration into a wide range of network devices and high performance test equipment.
Some zero reference algorithms are based on analysis of the decoded video stream, and the identification of visual impairments, and have been implemented by companies such as Genista. This approach is more complex and does require access to the decoded video stream.
There is standardization activity within the industry related to the definition and reporting of zero reference performance metrics - this is at an early stage, but is providing some structure within which better definition can be achieved. The approach under development (in parallel within several standards committees [10]) has the following metrics defined:
Some companies, such as Ineoquest, market technology that reports packet loss and jitter on the basis that if packet loss and jitter are low then quality must be ok. This approach would be fine in a scenario where quality was either perfect or terrible, but is likely to be less dependable when packet loss rates are “noticeable” and it becomes important to understand the impact of loss on the specific codec type and configuration.
Partial Reference Algorithms
Partial reference or reduced reference algorithms can also be used for in-service monitoring as they reduce the complexity of the real time analysis required.
Summary
Emerging video performance monitoring technology can be immensely helpful in testing video equipment performance, performing pre-deployment testing and in-service monitoring [11]. There is considerable activity within the industry in the development of new tools and technology for both full and zero reference video performance measurement.
References
[1] Digital Video Quality, Stefan Winkler, Wiley 2005
[2] Color Moving Pictures Quality Metric, van den Lambrecht, C J. PhD Thesis, EPFL 1996
[3] Spatial-temporal distortion metrics for in-service quality monitoring of any digital video system. Wolf, S., Pinson, M. Proc SPRI Multimedia SYstems, 1999
[4] A metric for continuous quality evaluation of compressed video with severe distortion, Masry, M., Hemami, S., Signal Processing: Image Communication 2004
[5] Final Report from the Video Quality Experts Group on the validation of objective models of video quality assessment. VQEG 2000
[6] ITU-T Recommendation J.144-2004, Objective Perceptual Video Quality Measurement Techniques for Digital Cable Television in the presence of a Full Reference.
---
A recognized authority in QoS and Packet Voice research and development, Alan Clark is the founder of Telchemy Incorporated, the inventor of the V.42bix data compression algorithm, and architect and editor of the V.58 network management standard.
|