Voip Performance Management

Transcript

VoIP Performance Management Alan Clark CEO, Telchemy 1 Internet Telephony Fall 2005 Internet Telephony - Fall 2005 Outline • • • • • 2 Problems affecting VoIP performance Tools for Measuring and Diagnosing Problems Protocols for Reporting QoS VoIP Performance Management Architecture Application to Enterprise and Service Provider Networks Internet Telephony Fall 2005 Enterprise VoIP Application IP Phone IP Phones IP VPN Branch Office Teleworker Gateway IP Phone 3 Internet Telephony Fall 2005 Residential VoIP Application PSTN Internet Residential Gateway 4 Internet Telephony Fall 2005 Trunking Gateway Potential Issues IP Phones Route flapping, Link failures, Delay IP VPN CODEC distortion Gateway LAN congestion, Long Ethernet segments, Duplex mismatch 5 Acoustic ECHO Line Echo Access Link Congestion Internet Telephony Fall 2005 IP Phone Call Quality Problems • • • • • • • 6 Packet Loss Jitter (Packet Delay Variation) Codecs and PLC Delay (Latency) Echo Signal Level Noise Level Internet Telephony Fall 2005 Packet Loss and Jitter Jitter Buffer IP Network Codec Distorted Speech Packets lost in network 7 Packets discarded due to jitter Internet Telephony Fall 2005 Jitter measurements can be misleading!!! 150 Average jitter level (PPDV) = 4.5mS Peak jitter level = 60mS Delay (mS) 125 100 75 50 0 0.5 1 Time (Seconds) 8 Internet Telephony Fall 2005 1.5 2 WiFi can also cause jitter 300 Recvd Signal Strength Delay (mS) & RSSI 250 Delay 200 150 100 50 7 5 1 0 0 1 2 5 1 5 0 1 7 5 2 0 0 2 2 5 2 5 0 2 7 5 3 0 0 3 2 5 3 5 0 3 7 5 4 0 0 4 2 5 4 5 0 5 0 2 5 0 0 Time 9 Internet Telephony Fall 2005 Effects of Jitter • Low levels of jitter absorbed by jitter buffer • High levels of jitter – lead to packets being discarded – cause adaptive jitter buffer to grow - increasing delay but reducing discards • If packets are discarded by the jitter buffer as they arrive too late they are regarded as “discarded” • Simple jitter metrics such as PPDV can be misleading 10 Internet Telephony Fall 2005 Packet Loss 500mS Avge Packet Loss Rate 50 Average packet loss rate = 2.1% Peak packet loss = 30% 40 30 20 10 0 30 35 40 45 50 55 Time (seconds) 11 Internet Telephony Fall 2005 60 65 70 Example Packet Loss Distribution los s 150 Co ns ec ut ive Burst weight (packets) 200 100 50 0 0 100 20 rst) u b e pars s ( y t i ens d t s r t bu n e c r pe 200 300 Burst length (packets) 12 Internet Telephony Fall 2005 400 500 Loss and Discard • Loss is often associated with periods of high congestion • Jitter is due to congestion (usually) and leads to packet discard • Hence Loss and Discard often coincide • Other factors can apply - e.g. duplex mismatch, link failures etc. 13 Internet Telephony Fall 2005 Example Loss/Discard Distribution los s 150 Co ns ec ut ive Burst weight (packets) 200 100 rce e p 25 rst d u b nt sp a ( y t i ens 50 0 0 100 200 300 Burst length (packets) 14 r t) s r u se b Internet Telephony Fall 2005 400 500 Band w id th (kb it/ s ) Leads To Time Varying Call Quality 500 400 300 200 100 0 5 High jitter/ loss/ discard Voice Data 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 MOS 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Time 15 Internet Telephony Fall 2005 Packet Loss Concealment Estimated by PLC algorithm • Mitigates impact of packet loss/ discard by replacing lost speech segments • Very effective for isolated lost packets, less effective for bursty loss/discard • But isn’t loss/discard bursty? – Need to be able to deal with 10-20-30% loss!!! 16 Internet Telephony Fall 2005 Effectiveness of PLC 5 Codec distortion G.711 no PLC G.711 PLC ACR MOS 4 Impact of loss/ discard and PLC 3 2 1 0 5 10 15 20 Packet Loss/Discard Rate Typical burst packet Loss/discard rate 17 Internet Telephony Fall 2005 Call Quality Problems • • • • • • • 18 Packet Loss Jitter (Packet Delay Variation) Codecs and PLC Delay (Latency) Echo Signal Level Noise Level Internet Telephony Fall 2005 Effect of Delay on Conversational Quality 5 M O S S core 4 3 2 Low echo level (55DB) Significant echo level (35dB) 1 0 100 200 300 400 Round trip delay (milliseconds) 19 Internet Telephony Fall 2005 500 600 Interaction of echo and delay • Echo with very low delay sounds like “sidetone” • Echo with some delay makes the line sound hollow • Echo with over 50mS delay sounds like…. Echo • Echo Return Loss – 55dB or above is good – 25dB or below is bad 20 Internet Telephony Fall 2005 Cause of Echo Gateway IP Echo Canceller Round trip delay - typically 50mS+ Acoustic Echo Line Echo Additional delay introduced by VoIP makes existing echo problems more obvious 21 Internet Telephony Fall 2005 Causes of Delay Accumulate and encode Echo Control CODEC RTP External delay IP UDP TCP Network delay Jitter buffer, decode and playout RTP IP UDP TCP 22 CODEC Echo Control Internet Telephony Fall 2005 Call Quality Problems • • • • • • • 23 Packet Loss Jitter (Packet Delay Variation) Codecs and PLC Delay (Latency) Echo Signal Level Noise Level Internet Telephony Fall 2005 Signal Level Problems Amplitude Clipping occurs -- speech sounds loud and “buzzy” 0 dBm0 -36 dBm0 Temporal Clipping occurs with VAD or Echo Suppressors -- gaps in speech, start/end of words missing 24 Internet Telephony Fall 2005 Noise • Noise can be due to – – – – Low signal level Equipment/ encoding (e.g. quantization noise) External local loops Environmental (room) noise • From a service provider perspective - how to distinguish between – room noise (not my problem) – Network/equipment/circuit noise (is my problem) 25 Internet Telephony Fall 2005 Measuring VoIP performance VoIP Specific Active Test - Measure test calls Passive Test - Measure live calls 26 VQmon ITU P.VTQ VQmon ITU P.VTQ Internet Telephony Fall 2005 Analog signal based ITU P.862 (PESQ) ITU P.563 “Gold Standard” - ACR Test 4 3 2 2 • Speech material – Phonetically balanced speech samples 8-10 seconds in length – Test designed to eliminate bias (e.g. presentation order different for each listener) – Known files included as anchors (e.g. MNRU) • Listening conditions – Panel of listeners – Controlled conditions (quiet environment with known level of background noise) 27 Internet Telephony Fall 2005 Example ACR test results • Extract from an ITU subjective test • Mean Opinion Score (MOS) was 2.4 • • • • • 28 1=Unacceptable 2=Poor 3=Fair 4=Good 5=Excellent 50 40 Votes 30 20 10 0 1 2 3 4 Opinion Score Internet Telephony Fall 2005 5 Measuring VoIP Performance • VQmon – Most widely used algorithm for VoIP performance monitoring. Fast efficient, supports narrow and wideband codecs, listening and conversational quality. Incorporates P.VTQ and G.107 as subsets, original model for RTCP XR. • ITU P.VTQ – In development - expected completion in mid-2006. Lightweight algorithm for narrowband use, currently only listening quality, may extend to conversational. • ITU G.107 E Model – Network planning tool, used as a basis for some monitoring applications. Inaccurate under conditions of bursty packet loss. • P.862 – Intrusive speech quality algorithm. Slow - takes a PC to process one speech file in approx real time. • P.563 – Non-intrusive algorithm that operates on analog speech data. Highly MIPS/Memory intensive and very inaccurate for individual calls. 29 Internet Telephony Fall 2005 Reminder - loss/jitter are time varying 150 Average jitter level (PPDV) = 4.5mS Peak jitter level = 60mS Delay (mS) 125 100 75 50 0 0.5 1 Time (Seconds) 30 Internet Telephony Fall 2005 1.5 2 VQmon algorithm 4 State Markov Model Gather detailed packet loss info in real time Arriving packets Loss/ Discard events Discarded Jitter buffer CODEC 31 Signal level Noise level Echo level Delay Metrics Calculation Internet Telephony Fall 2005 Call Quality Scores Diagnostic Data Modeling transient effects Ie(burst) Measured Call quality User Reported Call quality Ie(VQmon) Ie(gap) 10 32 15 20 25 Time (seconds) Internet Telephony Fall 2005 30 35 Computational model Burst loss rate Perceptual model Calculate R-LQ MOS-LQ Ie mapping Gap loss rate ETSI TS101 329-5 33 Signal level Noise level Calculate Ro, Is Echo Delay Calculate Id Recency model Modified ITU-T G.107 Calculate R-CQ MOS-CQ Internet Telephony Fall 2005 Accuracy: Non-bursty conditions Comparison of VQmon vs ACR MOS - ILBC 15.2k 5 4 ACR MOS 4.5 PESQ Score 3.5 3 2.5 2 VQmon MOS-PQ 3 2.5 2 1.5 1.5 1 PESQ 3.5 VQmon MOS-LQ 4 MOS Score Comparison of VQmon vs PESQ - ILBC 15.2k 1 0 5 10 15 20 0 5 Packet Loss Rate (%) 34 10 15 20 Packet Loss Rate (%) Internet Telephony Fall 2005 25 30 Comparison of VQmon and E Model VQmon – – – – – – • Extended G.107 Transient impairment model Wide range of codec models Narrow & Wideband Jitter Buffer Emulator Listening and Conversational Quality 3.5 3 V 2.5 on m Q E d Mo el 2 G.107 – Well established model for network planning – No way to represent jitter – Few codec models – Inaccurate for bursty loss – Conversational Quality only 35 4 Estimated MOS • 1.5 1.5 2 2.5 3 3.5 ACR MOS Comparison of VQmon and E Model for severely time varying conditions Internet Telephony Fall 2005 4 ITU P.563 - Passive monitoring • • Analyses received speech file (single ended) 5.00 Produces a MOS score • Correlates well with MOS when averaged over many calls ACR MOS 4.00 3.00 2.00 • Requires 100MIPS per call 1.00 • NOT suitable for individual calls 1 2 3 4 P563 Score Comparison of P.563 estimated MOS scores with actual ACR test scores. Each point is average per file ACR MOS with 16 listeners compared to P.563 score 36 Internet Telephony Fall 2005 5 Active or Passive Testing? • Active testing – works for pre-deployment testing and on-demand troubleshooting • But!!!! – IP problems are transient • Passive monitoring – Monitors every call made - but needs a call to monitor – Captures information on transient problems – Provides data for post-analysis • Therefore - you need both 37 Internet Telephony Fall 2005 VoIP Performance Management Framework Network Management System Call Server and CDR database Signaling Based QoS Reporting Network Probe, Analyzer or VQ Router VoIP Endpoint SNMP Reporting VQ VQ VoIP Gateway RTP stream (possibly encrypted) Embedded Monitoring 38 Media Path Reporting (RTCP XR) Internet Telephony Fall 2005 Embedded Monitoring RFC3611 - RTCP XR Loss Rate 39 Discard Rate Burst Density Gap Density Burst Duration (mS) Gap Duration (mS) Round Trip Delay (mS) End System Delay (mS) Signal level RERL Noise Level Gmin R Factor Ext R MOS-LQ MOS-CQ Rx Config - Jitter Buffer Nominal Jitter Buffer Max Jitter Buffer Abs Max Internet Telephony Fall 2005 RTCP XR Application “B” “A” Residential Subscriber RTCP XR RTCP XR Quality of stream from B to A and acoustic echo at A (if known) Quality of stream from A to B and Echo level on trunk side 40 Internet Telephony Fall 2005 Trunk side SIP Service Quality Reporting Event PUBLISH sip:[email protected] SIP/2.0 Via: SIP/2.0/UDP pc22.example.com;branch=z9hG4bK3343d7 ……… Content-Type: application/rtcpxr Content-Length: ... VQSessionReport LocalMetrics: TimeStamps=START:10012004.18.23.43 STOP:10012004.18.26.02 SessionDesc=PT:0 PD:G.711 SR:8000 FD:20 FPP:2 PLC:3 SSUP:on [email protected] ……… Signal=SL:2 NL:10 RERL:14 QualityEst=RLQ:90 RCQ:85 EXTR:90 MOSLQ:3.4 MOSCQ:3.3 QoEEstAlg:VQMonv2.1 DialogID:38419823470834;to-tag=8472761;from-tag=9123dh311 41 Internet Telephony Fall 2005 SIP QoS Reporting Application “B” “A” Trunk side Residential Subscriber SIP QoS SIP QoS report sent at end of call. Can report on both directions if RTCP XR is present in both endpoints, otherwise only on received direction. 42 Internet Telephony Fall 2005 “Collector” Enterprise Application VQ VQ IP Phone IP Phones VoIP SL A VQ VQ VQ VQ VQ VQ VQ SNMP NMS VQ Gateway IP VPN RT CP Branch Office Teleworker XR SIP QoS Report VQ VQ IP Phone 43 Internet Telephony Fall 2005 Actual (typical?) VoIP SLA Jitter < 20mS Loss < 0.1% per month Latency < 100mS Availability 99.9% What does this mean in practice? 44 Internet Telephony Fall 2005 A Better VoIP SLA 99.9% of calls/intervals have MOS-LQ > 3.9 MOS-CQ > 3.8 Degraded Service Quality Events < 0.1/ hour Based on either reference or actual endpoint Transient quality problems [DSQ = ….] Latency < 100mS Availability 99.9% 45 Internet Telephony Fall 2005 Also reflected in MOS-CQ Availability of media AND Signaling path Enterprise Applications • Ensure network is VoIP ready before deployment!! • Use VQmon/RTCP XR/ SIP QoS in IP phones and gateways • Use passive monitoring on every call to catch transient problems for post analysis • Develop meaningful SLAs 46 Internet Telephony Fall 2005 Residential VoIP Application VQ VQ PSTN VQ Internet CP T R VQ XR SIP QoS VQ Residential Gateway 47 Internet Telephony Fall 2005 Trunking Gateway Residential VoIP Application VQ VQ PSTN Internet VQ Trunking Gateway VQ CP T R XR SIP QoS Residential Gateway 48 Internet Telephony Fall 2005 Residential VoIP Management • Use RTCP XR between IP endpoints to provide more detailed call quality metrics and bidirectional reporting • Use SIP QoS reports to get data back to management systems • Insist that peer networks (either VoIP or PSTN) support RTCP XR and defined SLAs 49 Internet Telephony Fall 2005 Summary • • • • 50 Problems affecting VoIP performance Tools for Measuring and Diagnosing Problems Performance Management Architecture Applications Internet Telephony Fall 2005