Best Practices For Handset Testing With Spirent Communications

Transcript

Best Practices for Handset Testing with Spirent Communications Nomad Spirent Communications 7340 Executive Way Suite A Frederick, MD 21704 Document Version: 2.2 Best Practices for Handset Testing with Spirent Communications Nomad Contents 1 Introduction ..................................................................................................................................................................... 3 2 Time Synchronization .................................................................................................................................................. 3 3 Optimal Downlink Volume (Input Level) ............................................................................................................. 5 4 Optimal Uplink Volume (Output Level) ................................................................................................................ 6 5 High Speed Internet Connectivity ........................................................................................................................... 8 6 Handset Placement and Settings ............................................................................................................................. 8 7 Phone Number Entry .................................................................................................................................................... 9 8 Caller ID Block ............................................................................................................................................................... 10 9 Redial After a Drop ...................................................................................................................................................... 11 10 Best Practices for Mobile Terminated Call Performance Testing ....................................................... 12 Appendix A: Nomad Troubleshooting Techniques .................................................................................................. 13 Downlink Data Missing................................................................................................................................................... 13 Uplink Data Missing ......................................................................................................................................................... 14 Unsuccessful Online Merge ........................................................................................................................................... 15 Bluetooth The Bluetooth word mark and logos are owned by the Bluetooth® SIG, Inc. and any use of such marks by Spirent Communications, Inc. is under license. Other trademarks and trade names are those of their respective owners. Copyright © Spirent Communications, Inc. 2013 2 Best Practices for Handset Testing with Spirent Communications Nomad 1 Introduction In order to obtain the most accurate results from the Nomad system, Spirent Communications has compiled a set of Best Practices to follow during testing. The Best Practice categories are listed below along with the applicable test types: Best Practice Time Synchronization Optimal Downlink Volume (Input Level) Optimal Uplink Volume (Output Level) High Speed Internet Connectivity Handset Placement and Settings Phone Number Entry Caller ID Block Redial After a Drop Best Practices for Mobile Terminated Call Performance Testing Voice Quality • • • • • • • • Call Performance • • • • Each of the Best Practices described are followed by Spirent Communications engineers during field testing, and are recommended for all users of the system. For additional guidance in resolving issues with the Nomad system, please see Appendix A of this document which addresses troubleshooting techniques. 2 Time Synchronization Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the audio waveform must be centered within the playback window in order for Nomad to accurately calculate MOS. The two waveforms below are both clipped. The first waveform is clipped on the left. The second waveform is clipped on the right. Nomad will not be able to score either sample accurately and will display a NULL value. Figure 2-1 - Nomad Waveform Clipped on the Left Copyright © Spirent Communications, Inc. 2013 3 Best Practices for Handset Testing with Spirent Communications Nomad Figure 2-2 - Nomad Waveform Clipped on the Right The illumination of the MOS statistics area in red also indicates this problem. This waveform is centered within the window and MOS will be calculated correctly: Figure 2-3 - Centered Waveform Synchronize the test laptop’s clock with the Audio Server to ensure consistent timing between the two:    During testing, select the Time Sync button. Set the Time Synchronization Source as either Internet Time Server or GPS. o Note: Spirent does not recommend using Internet Time Server synchronization for Voice Quality testing performed while drive testing. Substantial latency may be introduced when the internet time protocol operates in a mobile environment. If possible, use GPS time synchronization in this case. Click the Sync Now button. Enable automatic time synchronization to prevent time drift in the synchronization between the test laptop and the Audio Server:    In the Time Sync dialog, check Enable automatic time synchronization. Set the period between time synchronization updates in minutes. Spirent recommends setting this to 5 minutes for frequent updates. Copyright © Spirent Communications, Inc. 2013 4 Best Practices for Handset Testing with Spirent Communications Nomad 3 Optimal Downlink Volume (Input Level) Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the downlink waveform must be completely visible within the playback window and free of noise distortion for Nomad to accurately calculate downlink MOS. When using analog device connections, Spirent recommends running the Auto-Level Assist Wizard prior to testing a new device to determine the optimal volume settings for that device. Minor adjustments to the Wizard recommendations can be made during testing as described below. The Downlink Volume (Input Level) for this audio sample is too high. The top and bottom of the waveform are clipped and distortion is present. Nomad will not be able to score this sample accurately: Figure 3-1 - High Downlink Volume An additional indicator of clipping, the Overdrive light for the saturated channel blinks red on the Spirent ME hardware unit and on the Levels tab of the Settings dialog: Figure 3-2 - Overdrive Indicator Illuminated This sample represents a channel set at the appropriate Downlink Volume. The waveform is intact and downlink MOS will be calculated correctly: Copyright © Spirent Communications, Inc. 2013 5 Best Practices for Handset Testing with Spirent Communications Nomad Figure 3-3 - Appropriate Downlink Volume To find the optimal Downlink Volume for any channel:       Set the handset volume one level below the maximum. Further volume adjustments will be performed using the Nomad software. While a test session is in progress, click the Settings button and navigate to the Levels tab. For the channel of interest, move the Input slider to the far right until the Overdrive light illuminates in the GUI and on the hardware unit. Move the Input slider to the left one notch at a time and observe the Overdrive light during the Recording state. The first position at which the Overdrive light stays off for the current and next recording cycle should be used as the Input setting. The optimal Input slider position should fall within the Normal Operating Range of 170225 shown on the slider. The downlink insertion gain (PESQ) or attenuation (POLQA) shown in the test window will likely read around 0. Note that high downlink volume will not be an issue when using a Bluetooth connection. In cases where the downlink insertion gain (PESQ) or attenuation (POLQA) for a Bluetooth channel is far from 0 (roughly less than -8 dB for insertion gain or greater than 8 dB for attenuation), the Input Level may be adjusted by moving the slider one or two notches to the right. The slider should be kept within the Normal Operating Range. If MOS deteriorates after the adjustment, return the Input Level to the original value. 4 Optimal Uplink Volume (Output Level) Nomad calculates MOS in Voice Quality tests by comparing audio samples degraded by the communication channel to the original source sample. Therefore, the uplink waveform must be free of noise distortion in order for Nomad to accurately calculate uplink MOS. When using analog device connections, Spirent recommends running the Auto-Level Assist Wizard prior to testing a new device to determine the optimal volume settings for that device. Minor adjustments to the Wizard recommendations can be made during testing as described below. Copyright © Spirent Communications, Inc. 2013 6 Best Practices for Handset Testing with Spirent Communications Nomad The uplink speech waveform is not displayed during testing. In most cases, uplink volume adjustment is only necessary if uplink MOS values are unexpectedly poor. The MOS Trend Window in this sample presents symptoms of an uplink volume setting that is too high: Figure 4-1 - High Uplink Volume This uplink MOS trend in this sample represents a channel set at an appropriate uplink volume: Figure 4-2 - Appropriate Uplink Volume To find the optimal Uplink Volume for any channel in both the Bluetooth and analog connection cases:    While a test session is in progress, click the Settings button and navigate to the Levels tab. Move the Output slider for the channel of interest within the Normal Operating Range of 120-175. If uplink MOS deteriorates after the adjustment, return the Output Level to the original value. Copyright © Spirent Communications, Inc. 2013 7 Best Practices for Handset Testing with Spirent Communications Nomad 5 High Speed Internet Connectivity Nomad requires a sustained internet connection of at least 50 kbps in the following situations:      Determining the optimal volume levels for a Voice Quality test using Automatic Level Assist. Viewing live uplink data during a Voice Quality test. Starting a Mobile Terminated call campaign. Obtaining the report for a Mobile Terminated or Voice Quality call campaign. Automatic internet-based time sync during testing (GPS time sync is recommended). Follow these guidelines for the best results when using internet-dependent features:     A sustained connection of 50 kbps is required to run these features. If the internet connection is lost during testing, these features will become unavailable. Functionality will return when the connection comes back up. The internet connection may need to be restarted manually. If possible, Spirent recommends using a different provider for the high speed internet connection than for the voice network. Spirent recommends using the Auto Level Assist Wizard to optimize volume prior to testing in a location with good RF coverage and a strong internet connection. 6 Handset Placement and Settings Noise coupling can induce buzzing or an equivalent noise on the audio path and reduce MOS scores. Follow these guidelines regarding handset placement and settings to minimize noise coupling:        During testing, position the handsets as far away from the Spirent ME unit as possible. If an AC inverter is used, keep it away from both the hardware unit and the handsets. Ensure that the test PC is not running on batteries. Do not charge the handsets during testing. During testing, the handsets should be set to standard audio – not speakerphone. If a 2.5 mm adaptor is required, ensure that the correct adaptor has been selected and that all adaptor connections are secure. Prior to testing, disable all Bluetooth connections with the exception of those used to couple the handset to the hardware unit. Copyright © Spirent Communications, Inc. 2013 8 Best Practices for Handset Testing with Spirent Communications Nomad 7 Phone Number Entry When performing Voice Quality testing using the Audio Server, it is imperative that the Number to call and Number calling from phone numbers match the actual numbers used. This ensures that uplink data may be obtained from the server and merged with the corresponding downlink data. Nomad provides the Connection Assistant utility to assist in entering the phone numbers with the highest accuracy. While Spirent strongly recommends the use of this utility for all customers, international users outside of North America are urged to take advantage of the Connection Assistant to ensure that phone numbers are entered in the correct format. To use the Connection Assistant:     On the Test Setup tab, select the Session button for any configured Mobile task. Confirm that the Uplink Device is set to Audio Server. Click the Connection Assistant button to start the utility. Click the Edit Audio Server Settings button. In the Edit Audio Server Settings dialog, select the Audio Server Name and enter the Username and Password. Click OK to accept the changes. Figure 7-1 - Edit Audio Server Settings Dialog      Dial the Audio Server using the handset to be connected to this channel. The handset does not need to be connected at this time. All active calls to the Audio Server will be represented in the Connection Assistant dropdown window using the {4 Digit DNIS}-{10 Digit ANI} convention: o {4 Digit DNIS} – The last four digits of the Audio Server phone number called by the mobile. o {10 Digit ANI} – The phone number of the mobile being tested. If the test handset does not appear in the list, use the Refresh button to update. It may take up to one minute for the handset to appear. Use the drop-down to select the correct number combination from the list. Click OK. Copyright © Spirent Communications, Inc. 2013 9 Best Practices for Handset Testing with Spirent Communications Nomad Figure 7-2 - Connection Assistant  The Number to call and Number calling from fields will be populated based on your selection, removing the risk of manually entering this information incorrectly. Figure 7-3 - Phone Number Settings Populated by the Connection Assistant 8 Caller ID Block Nomad relies on the phone numbers of both the calling and called devices to identify and merge session data. Nomad will not be able to correlate session uplink data if Caller ID Block is turned on. To this end, it is critical that Caller ID Block is turned OFF for all test devices. Copyright © Spirent Communications, Inc. 2013 10 Best Practices for Handset Testing with Spirent Communications Nomad 9 Redial After a Drop Unlike most RF collection tools, Nomad channels must be configured by the user in order to automatically redial after a dropped call during Voice Quality testing. Auto redial is available for all devices using a Bluetooth connection and for some devices using an analog connection. Bluetooth auto redial may be enabled via the Channel Settings button for any configured Voice Quality task. If auto redial is not enabled, the handset must be manually redialed before proceeding with testing after a drop. This is important to keep in mind during long call testing when calls are allowed to stay connected until they drop. To enable auto redial:   In the case of a Bluetooth connection: o Auto redial is available for all devices. o In the Channel Settings dialog, set the Audio Interface to Bluetooth. o Check the Enable Bluetooth auto redial box and specify the number of seconds to pause before redialing. o With this configuration, a call that is dropped during testing will be automatically redialed. In the case of an analog connection: o Auto redial is available for phones which support the following options:  Ability to maintain an analog and Bluetooth connection simultaneously.  Ability to transfer from a Bluetooth connection to an analog connection.  Ability for the Bluetooth connection to be re-established while the phone is on a call. o To utilize auto redial with a qualified device:  Connect the test handset to the Spirent ME hardware using Bluetooth.  Next, attach the test handset to the same channel using an analog connection.  Verify that the Bluetooth connection is still active.  Set the Audio interface for this channel as Analog.  Check the Enable Bluetooth auto redial option.  Specify the number of seconds to Pause before redialing.  With this configuration, a call that is dropped during testing will be redialed using the Bluetooth connection. After three cycles of silence, Nomad will then issue a command to transfer the audio from the Bluetooth to the analog connection. Figure 9-1 - Auto Redial Option in Channel Settings Dialog Copyright © Spirent Communications, Inc. 2013 11 Best Practices for Handset Testing with Spirent Communications Nomad 10 Best Practices for Mobile Terminated Call Performance Testing In a Mobile Terminated Call Performance Test, the Call Server makes calls to the mobile test device for call control testing. Statistics gathered during this type of test are based on the point of view of the test handset. Spirent recommends following these best practices for the best results with Mobile Terminated testing:    Confirm that the test handset is properly configured with voicemail. This will ensure that voicemail is properly detected during testing. The Access Timeout parameter for a Mobile Terminated call campaign must be set to a larger duration than it takes the phone to go to voicemail. If Access Timeout is too short, Nomad will fail to recognize a call sent to voicemail. Instead, such a call will be classified as an Access Timeout before voicemail picks up. A 45-second Access Timeout is generally sufficient for North American carriers. Nomad can take up to seven seconds to detect voicemail. In a Synchronous campaign it is best practice to set Duration at least seven seconds longer than Access Timeout to ensure that no Voicemail events are missed. Copyright © Spirent Communications, Inc. 2013 12 Best Practices for Handset Testing with Spirent Communications Nomad Appendix A: Nomad Troubleshooting Techniques This Appendix provides techniques for troubleshooting and resolving some of the most common issues encountered when testing with the Nomad System:    Downlink data missing (uplink data may be available) Uplink data missing (downlink data may be available) Unsuccessful online merge (data appears on Incomplete tab) If the troubleshooting tips presented here are insufficient, or if you encounter other issues, please contact Spirent Product Support for assistance. Downlink Data Missing In the event that downlink data is missing during Voice Quality testing (even though uplink data may be available), check these possible causes: Phone Volume   Is the phone volume level acceptable? Confirm that the handset volume is set one level below the maximum. Time Synchronization   Is the test laptop’s clock synchronized with the Audio Server? If performing the test while drive testing, is the GPS time synchronization method used? See Chapter 2 – Time Synchronization for instructions on checking and setting synchronization.  Input Level    Adapter Issues     Task Type / Corresponding Device   Is the Input Level properly set? Input Level refers to the volume of the sourced downlink audio sample. See Chapter 3 – Optimal Downlink Volume to make the proper adjustments. Is there a problem with the 2.5 mm adapter? Confirm that you are using an appropriate adapter for the device. Ensure that all adapter connections are secure. If necessary, try a different adapter to help isolate the source of the problem. Has the appropriate Task Type (i.e. Mobile, Base, Source Only, Record Only) been applied to this channel? In the Session dialog, is the Uplink, Downlink, Sourcing or Recording Device set as appropriate? Copyright © Spirent Communications, Inc. 2013 13 Best Practices for Handset Testing with Spirent Communications Nomad Uplink Data Missing In the event that uplink data is missing during Voice Quality testing (even though downlink data may be available), check these possible causes: Internet Access      Phone Numbers   Does the test laptop have a sustained high-speed internet connection of at least 50 kbps? If the internet connection was lost during testing, live uplink data will become unavailable. Functionality will return when the connection comes back up. In some cases, the internet connection may need to be restarted manually. See Chapter 5 – High Speed Internet Connectivity for more information on this topic. On the Test Setup tab, open the Session dialog for the affected channel and confirm that the Phone Number Settings are correct. Spirent recommends using the Connection Assistant to confirm phone numbers as described in Chapter 7 – Phone Number Entry. Phone Volume   Is the phone volume level acceptable? Confirm that the handset volume is set one level below the maximum. Time Synchronization   Is the test laptop’s clock synchronized with the Audio Server? If performing the test while drive testing, is the GPS time synchronization method used? See Chapter 2 – Time Synchronization for instructions on checking and setting synchronization.  Output Level    Is the Output Level properly set? Output Level refers to the volume of the sourced uplink audio sample. See Chapter 4 – Optimal Uplink Volume to make the proper adjustments. Adapter Issues     Is there a problem with the 2.5 mm adapter? Confirm that you are using an appropriate adapter for the device. Ensure that all adapter connections are secure. If necessary, try a different adapter to help isolate the source of the problem. Task Type / Corresponding Device  Has the appropriate Task Type (i.e. Mobile, Base, Source Only, Record Only) been applied to this channel? In the Session dialog, is the Uplink, Downlink, Sourcing or Recording Device set as appropriate? Caller ID Block   Confirm that Caller ID Block is turned OFF for the test device. Copyright © Spirent Communications, Inc. 2013 14 Best Practices for Handset Testing with Spirent Communications Nomad Unsuccessful Online Merge On occasion, Nomad may be unsuccessful in merging the uplink and downlink data from a test session into a unified log file for report generation. Such files will appear on the Data → Incomplete tab at the conclusion of a test (successfully merged files appear on the Data → Complete tab). Files on the Incomplete tab contain an icon signifying each problematic channel:  The half-circle icon indicates that Nomad does not have access to all of the data required to merge this file. This might appear if no calls were started for a channel during a test. This might also appear in the case of a Remote Unit test where Nomad can access the locally collected data but not the remotely collected data.  The yellow triangle icon indicates that an error has occurred during data collection preventing Nomad from merging the uplink and downlink data associated with this test case. One likely error cause is the incorrect entry of the Phone Number Settings. Another common cause of this error is lack of internet connectivity at merge time. Click on the or troubleshoot: indicator icon to invoke the Modify Incomplete Logging Session dialog to Retry Auto-Complete  If lack of internet connectivity is believed to be the reason for the error, re-establish the internet connection and then use the Retry Auto-Complete button to merge the data and move to the Complete tab. Phone Number Settings     Uplink Scoring Utility   Mark As Complete   If the icon indicates Unable to download uplink scores, the Phone Number Settings for the session are likely incorrect. Select the Edit button to Edit Channel Configuration. Enter the correct values for Number to call and Number calling from and then Close the dialog. Click Retry Auto-Complete to merge the data. Some files that are unable to be merged automatically can still be merged using the Uplink Scoring Utility. Contact Spirent if you require access to your uplink data for merging. If you believe that a channel has been flagged as incomplete in error, use the Mark As Complete option to disregard the incomplete data warning. Any report generated with this data may contain missing or incomplete information. Copyright © Spirent Communications, Inc. 2013 15