Firn User Guide

Transcript

FIRN User Guide A Guide to assist customers with the use of services offered under the FIRN contract. Version 9 21 14 TABLE OF CONTENTS 1 OVERVIEW ......................................................................................................................................................................... 4 1.1 1.2 1.3 2 FIRN SERVICE OVERVIEW .............................................................................................................................................. 5 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3 TROUBLE MANAGEMENT SUPPORT ................................................................................................................................. 14 TROUBLE REPORTING PROCEDURES ................................................................................................................................ 14 CUSTOMER RESPONSIBILITIES ......................................................................................................................................... 14 HAYES NOC RESPONSIBILITIES ...................................................................................................................................... 17 MFN NOC RESPONSIBILITIES ......................................................................................................................................... 17 TROUBLE TICKET STATUS CHECK ................................................................................................................................... 18 TROUBLE TICKETS CLASSIFICATIONS.............................................................................................................................. 18 NOTIFICATION COMMITMENTS ........................................................................................................................................ 20 ESCALATION PROCEDURES.............................................................................................................................................. 21 GENERATING CONFIGURATION CHANGE REQUEST .................................................................................................... 24 CUSTOMER MANAGED OPTION .................................................................................................................................. 25 HOURS OF OPERATION................................................................................................................................................ 26 MFN NOC CONTACTS, ESCALATION LEVELS & RESPONSIBILITIES ........................................................................... 26 ORDERING FIRN SERVICES .......................................................................................................................................... 31 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 4.14 4.15 4.16 4.17 4.18 4.19 4.20 5 INTRODUCTION.................................................................................................................................................................. 5 KEY BENEFITS AND FEATURES.......................................................................................................................................... 5 TYPES OF SERVICES........................................................................................................................................................... 6 FIRN DAN (DISTRICT AREA NETWORK) .......................................................................................................................... 7 OPTIONAL SERVICES ....................................................................................................................................................... 11 SLA OVERVIEW .............................................................................................................................................................. 11 SLA CATEGORIES ........................................................................................................................................................... 12 SERVICE LEVELS EXCLUSIONS ........................................................................................................................................ 13 FIRN SLA APPLICABILITY – SERVICE TYPES ................................................................................................................. 13 FIRN CUSTOMER SUPPORT........................................................................................................................................... 14 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 4 FIRN MISSION STATEMENT .............................................................................................................................................. 4 FIRN NETWORK OPERATIONS GUIDE ............................................................................................................................... 4 AUDIENCE ......................................................................................................................................................................... 4 CUSTOMER ENGAGEMENT ............................................................................................................................................... 31 PREREQUISITES FOR ORDERING ....................................................................................................................................... 34 INITIAL SERVICE CSAB ORDER SUBMISSION – INSTALLS ............................................................................................... 36 IN-HOUSE RELOCATIONS ................................................................................................................................................ 36 RELOCATION OF SERVICE TO NEW PHYSICAL ADDRESS ................................................................................................. 36 CHANGES TO EXISTING SERVICE ..................................................................................................................................... 37 FIRN MANAGED OPTION ................................................................................................................................................ 37 CUSTOMER MANAGED OPTION ....................................................................................................................................... 38 CHANGES REQUIRING A CSAB NUMBER CHANGE .......................................................................................................... 39 DISCONTINUATION OF SERVICE .................................................................................................................................. 39 MANDATORY USE OF ONLINE CSAB SYSTEM ........................................................................................................... 39 INCOMPLETE CSAB ORDERS...................................................................................................................................... 39 FIRN INSTALLATION PROCESS ................................................................................................................................... 40 INSTALLS, MOVES, ADDS, AND CHANGES (IMAC) SLAS .......................................................................................... 41 SLA INSTALLATION PERFORMANCE TARGETS ........................................................................................................... 42 OTHER SERVICES AND IMPACT ON SLA PERFORMANCE TARGET ............................................................................... 43 CSAB ORDER SUBMITTAL AND SLA PERFORMANCE TARGET FOR NEW INSTALLATIONS ........................................... 43 BOOK OF RECORD AND SLA HOLD TIME GUIDELINES ............................................................................................... 44 SLA HOLD TIME REQUEST AND APPROVAL PROCESS ................................................................................................ 44 SLA HOLD TIME REASONS......................................................................................................................................... 45 BILLING ............................................................................................................................................................................. 45 5.1 5.2 5.3 5.4 5.5 5.6 5.7 BILLING CYCLES ............................................................................................................................................................. 45 MINIMUM BILLING PERIODS ........................................................................................................................................... 46 BILLING START DATE FOR NEW SERVICES ....................................................................................................................... 46 CPE ACQUIRED THROUGH THE FIRN CONTRACT ............................................................................................................ 46 CUSTOMER PROVIDED CPE ............................................................................................................................................. 46 DISCONNECT PROCESSING AND EFFECTIVE BILL DATE................................................................................................... 46 CHANGE IN BILLING RESULTS FROM A FUNDING CHANGE PROVISO LANGUAGE BY THE LEGISLATURE, OR A RESPONSIBILITY CHANGE .......................................................................................................................................................... 47 5.8 CREDITS AND DISPUTE RESOLUTION GUIDELINES .......................................................................................................... 49 6 NMS TOOLS ...................................................................................................................................................................... 49 6.1 6.2 6.3 6.4 6.5 6.6 6.7 7 CUSTOMER PREMISE EQUIPMENT (CPE) .................................................................................................................. 57 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15 8 FIRN SECURE INTERNET BUNDLED SERVICES (SECURE INTERNET SERVICES) ............................................................... 67 FIRN ADVANCED SECURITY OFFERINGS (ASO) ............................................................................................................. 73 APPENDICES..................................................................................................................................................................... 81 10.1 10.2 10.3 10.4 10.5 10.6 10.7 11 DESIGN OVERVIEW ......................................................................................................................................................... 63 8.2 ROUTING PROTOCOL REQUIREMENTS ....................................................................................................................... 65 IP QOS REQUIREMENTS .................................................................................................................................................. 65 CPE & ACCESS METHOD REQUIREMENTS ...................................................................................................................... 66 ADDITIONAL SERVICES SUPPORTED THROUGH FIRN .......................................................................................... 67 9.1 9.2 10 FIRN CPE ....................................................................................................................................................................... 57 MAINTENANCE ................................................................................................................................................................ 58 INSTALLATION ................................................................................................................................................................ 58 ACQUIRED ....................................................................................................................................................................... 58 OUT OF BAND (OOB) ACCESS ........................................................................................................................................ 59 ROUTER CONFIGURATION BACKUP ................................................................................................................................. 59 ROUTER MANAGEMENT .................................................................................................................................................. 59 ROUTER ACCESS ACCOUNT MANAGEMENT ..................................................................................................................... 60 AFTER HOURS MAINTENANCE ACTIVITY ........................................................................................................................ 60 FIRN CPE SLA MATRIX ............................................................................................................................................ 60 FIRN CPE ROADMAP / MODELS ................................................................................................................................ 61 GENERAL.................................................................................................................................................................... 61 SPECIAL CPE NEEDS................................................................................................................................................... 61 NEW CPE MODELS ..................................................................................................................................................... 62 FIRN CPE NAMING CONVENTION METHODOLOGY ................................................................................................... 62 NETWORK ENGINEERING & DESIGN ......................................................................................................................... 63 8.1 8.2 8.3 8.4 9 NETWORK MANAGEMENT SYSTEMS CPE REQUIREMENTS FOR MONITORING................................................................. 49 MFN NETWORK MANAGEMENT TOOLS USER TRAINING ................................................................................................ 50 MFN NMS TOOLS ACCESS REQUESTS............................................................................................................................ 51 ROUTINE CHANGES ......................................................................................................................................................... 51 ROUTINE REMOVALS....................................................................................................................................................... 51 EMERGENCY ACCOUNT REMOVALS ................................................................................................................................ 51 NETWORK MANAGEMENT REPORTS ................................................................................................................................ 52 FIRN CORE LAYOUT .................................................................................................................................................. 81 FIRN TYPES OF ACCESS ............................................................................................................................................. 81 FIRN VS. FIRN DAN COMPARISON ........................................................................................................................... 81 NMS TOOLS ACCESS – CSAB AND ACCESS FORM SAMPLES ..................................................................................... 81 ORDER DESIGN REFERENCE DOCUMENTS .................................................................................................................. 81 CUSTOMER SERVICE QUESTIONNAIRE ........................................................................................................................ 81 SECURE INTERNET SERVICES QUESTIONNAIRES ......................................................................................................... 81 GLOSSARY OF TERMS ................................................................................................................................................... 83 12 REVISION HISTORY ........................................................................................................................................................ 84 1 Overview 1.1 FIRN Mission Statement The State of Florida, DMS, AT&T and its partners will provide timely, quality, and professional network support to the State of Florida for the FIRN Contract. 1.2 FIRN Network Operations Guide The FIRN Operations Guide is being developed jointly by DMS-SUNCOM and AT&T to describe the operational handling of the day-two support for FIRN Services. The Guide will provide detailed operational procedures and will show the relationship of the Operational Personnel and Network Management System (NMS) Tools within the FIRN NOC. This Guide is not intended to be a User Guide for NMS Tools and the specific guide for any tool referenced should be consulted for detailed information regarding the capabilities and use of that tool. This Operations Guide will be periodically reviewed by DMS-SUNCOM, AT&T, Embarq and Hayes staff and revisions will be implemented as necessary. Any proposed changes or updates will be approved by both AT&T and DMS-SUNCOM. The AT&T Life Cycle Manager will be the primary contact person for changes and revisions. From DMS-SUNCOM, the FIRN Product Manager will be the primary contact to coordinate these changes, revisions and approvals. Contact Telephone Email Role Connie Gruen 850-216-3515 [email protected] Daniel Bowen 850-216-3543 [email protected] Freddy Martinez 850-922-7475 [email protected] m AT&T Customer Care Life Cycle Manager AT&T Marketing Life Cycle Manager DMS FIRN Product Manager 1.3 Audience This document is to be used to govern and define the rules of engagement between DMS of the State of Florida and AT&T and its sub-contractors. This document will be in compliance with the FIRN contract which was signed on January 12, 2009. In the case of any conflict between this document and the contract, the Contract has the ruling authority. This document should be considered proprietary information between the State of Florida’ Department of Management services and AT&T and its subcontractors. This document is not to be shared in whole or part with any Agency personnel. 2 FIRN Service Overview 2.1 Introduction FIRN is a comprehensive solution providing a rich and flexible private enterprise communications infrastructure dedicated for the exclusive use of State of Florida eligible users. This enterprise infrastructure is based on a MultiProtocol Label Switching (MPLS) technology and Quality of Service mechanisms providing improved security and robust connectivity resulting in a highly available (HA) and highly reliable (HR) statewide communications network. The FIRN network infrastructure uses the existing MyFloridaNet network backbone as transport. Please refer to Appendix 14.3 for a layout of the MyFloridaNet core. FIRN will provide service elements such as network core, local loop access, customer premises equipment (CPE), security, Internet access, network management tools, design and engineering, billing and ordering as a complete turn-key solution with mostly flat-rate pricing statewide. Customers have the option to either manage CPE or let FIRN manage it. There is no additional cost to manage a CPE router if the FIRN Bundled Package is selected. The FIRN network is capable of transmitting over single user local loop connection application traffic such as internet, voice, data, and video. Any eligible educational entity that is a qualified SUNCOM user (customer) can subscribe to FIRN. The categories listed below can subscribe to FIRN services.         State Agencies with educational programs State Universities or Community Colleges Private, Non-profit Universities as defined in Florida State statutes FS 282.703(3) & FS 1009.89(3) Educational City or County Governments in Florida Private, Non-Profit Elementary and Secondary Schools as defined in Florida State statute FS 282.705 (5) Educational School Boards Libraries as defined in Florida State statute FS 282.706 DOE related entities which qualify as SUNCOM users. Care must be exercised when working with an eligible entity in that unless they are an educational and Erate entity some restriction may apply. For example, DMS offers services which may be directly in competition with this contract if an entity is not E-rateable and happens to be a State Agency. For these special cases all FIRN team members must consult with DMS before giving approval or the impression to the user entity that the service sought is a workable solution under the FIRN contract. There may be a different State of Florida contract that should be used. 2.2 Key Benefits and Features FIRN offers a variety of benefits with a superior solution that delivers: Core Nodes: The MyFloridaNet/FIRN core features dual carrier class Juniper M320 routers in each LATA-based node location statewide for a total of 20 M320 core routers. These powerful nodes are interconnected by a combination of redundant OC-48 (2.4 Gbps) and OC-12 (622 Mbps) packet over SONET links, providing a tremendous amount of bandwidth with full protection against core link failure. Internet: Internet access is equal to the access speed of the FIRN connection is included in the basic service. Firewall Services are available as options. Simplified Lower Pricing: A vast majority of the FIRN pricing is flat rate statewide. For instance, customers in Blountstown will now pay the same price for a T1 connection as the customers in downtown Miami. Proactive SLAs: FIRN includes Service Level Agreements. All SLAs are monitored 24 x 7 by the FIRN Web based NMS by the FIRN/MFN NOC. Any-to-Any Connectivity: FIRN features fully meshed connectivity between all sites without the cost and complexity of configuring multiple PVCs. MPLS supports scalable, any-to-any connectivity within customer specific VPNs. Customers are able to order new sites without ever having to make router configuration changes at any of their other locations. One Connection: FIRN utilizes a private MPLS based IP core to create an integrated network capable of carrying internet, intranet (private VPN) and Extranet traffic all on one connection, helping to cut connection, management and equipment costs. Network Operation Center: FIRN includes a dedicated world-class Network Management Solution and Network Operations Center located in Tallahassee as well as a live, mirrored, Network Management System and Network Operation Center in Winter Park, Florida. Network Management Tools Suite: The FIRN NOC will monitor services by utilizing state of the art management tools such as NetQoS Report Analyzer, and CA-Spectrum Service Performance Manager. Remedy trouble ticketing system and eHealth are also included as part of the tools suite and basic service. Access Choices: FIRN access method is determined by the service provider with the exception of DSL. Customers can request DSL service specifically. Access types include DSL, Frame Relay, and Ethernet Note: for details and applicability, please refer to the section on “Types of Service”. 2.3 Types of Services Pricing is for illustration purposes only. For most recent & updated pricing, please visit the FIRN web site. 2.3.1 Regular FIRN FIRN access connectivity originates at an individual user’s location and terminates at the Provider Edge router, creating an onramp to the FIRN core. FIRN customers will be provided the appropriate access (local loop) type based upon the specific applications and needs at any given site. For example, customers may have Frame Relay access at one location, Ethernet access at another, and DSL at yet another. Because FIRN provides interoperability across multiple access types, these locations will work together seamlessly. Please refer to Appendix 14.4 for a layout of the different types of access connecting into FIRN. FIRN access connectivity service is a requirement for ordering and/or retaining any FIRN Other Network Services (ONS). 2.3.1.1 Frame Relay Access Customers with frame relay access connect into the FIRN Core. When you are provided frame relay access you will be provided an access speed that is equal to your FIRN port speed, which includes full backbone access equal to your FIRN port speed. Frame Relay access characteristics are listed below.    Full CIR Bandwidth Range – 64kbps to 45Mbps SLA & QoS Support 2.3.1.2 Ethernet Access Customers can be provided Ethernet access to connect into the FIRN Core. For Ethernet access you will be provided an access speed that is equal to your FIRN port speed, and includes full backbone access that is equal to your FIRN port speed. Ethernet access characteristics are listed below.    Full access and port bandwidth Bandwidth Range – 2Mbps to 1Gbps SLA & QoS Support 2.3.1.3 DSL Access Customers can select DSL access to connect into the FIRN Core. When DSL service is selected access will be provided at a speed that is equal to the FIRN port speed, which includes full backbone access equal to the FIRN port speed. DSL access characteristics are listed below.    Best Effort Three Bandwidth Ranges No QoS Support, or Tools access 2.3.2 Pricing Structure Below is the link showing the pricing elements for FIRN service http://www.dms.myflorida.com/business_operations/telecommunications/suncom2/data_services/florida_informatio n_resource_network_firn The FIRN user must order a regular FIRN connection or access service before ordering any FIRN Other Network Services (ONS). Additionally, a FIRN connection must exist at all times to maintain ONS services. FIRN customers will not be allowed to disconnect the FIRN main connection and retain ONS services. In the event ONS are required and the FIRN connection is terminated, the FIRN user will have the option to order like services out of the regular SUNCOM offerings. Other SUNCOM services, unlike FIRN, may require and installation charge to initiate new services. 2.4 FIRN DAN (District Area Network) The (District Area Network) DAN is designed for local customers with no requirements for internet access The connection is basically a managed FIRN Bundled offering with no internet access. In this instance, Access is combined with a 4M FIRN Core port for management purposes only. Customers on the DAN can communicate with one another over a single multi-access VLAN and through a connection to the main District DAN router. The DAN customer gets limited features of regular FIRN (management, administration, etc.) with a reduced monthly cost, since no internet access is provided. Please refer to the diagram below of a sample configuration using FIRN DAN. DAN Design Requirements:        Host and remote routers must be provider managed. Host circuit BW must be greater than or equal to the cumulative total of the remote circuits BW. Network must be configured in a hub and spoke arrangement. No QoS SLAs. No customer provided CPE. Sites cannot be provisioned across LATA boundaries. o Out of LATA sites can be connected via a private VRF There will be a SPOC for trouble reporting (District IT staff). In addition to the single, multi-access VLAN for intra-DAN communications, DAN subscribers can also have access to the Internet using the District’s regular FIRN internet connection, DAN Ethernet Speed Location Management VRF Routing Protocol (usually BGP) Direct Access to Other Ethernet Sites Only For Access to Frame Relay Sites Must Route via Host DAN Ethernet Speed Location Management VRF Routing Protocol (usually BGP) Direct Access to Other Ethernet Sites Only For Access to Frame Relay Sites Must Route via Host e Eth rn e t Primary MGMT VLAN Secondary MGMT VLAN Access to Ethernet Sites Access To Frame Sites Ethernet et rn Et he Primary PE Secondary PE Host connection must be larger than 37.5 M MFN Core DAN HOST Site Management VRF Customer LAN FIRN Internet Access managed router at Host site District office 2.4.1 DAN Pricing Structure Refer to the table below listing the DAN price structure. This is the end user rate. E-Rate Bundled Services Distric Area Network (DAN) Pricing CSA FORMAT = AGY - $I RT - NNNN - YR - OC¹ Bandwidth² Frame Frame MetroE The host site may not be over subscribed – the aggregate bandwidth of all frame relay sites plus Ethernet sites may not exceed the Host’s connection speed. Example: 5 Frame sites at 1.5 M each 3 Ethernet sites at 10 M each 37.5 M Total Primary MGMT VLAN Secondary MGMT VLAN Access to Ethernet Sites Access To Frame Sites CSA RT Code DAN Frame Speed Location Management VRF Routing Protocol (usually BGP) Direct Access to Other Frame Sites Only For Access to Ethernet Sites Must Route via Host Monthly Recurring Rate 1.5 Mbps x 256 kbps - DSL N/A D1 384 kbps x 384 kbps - DSL N/A D2 768 kbps x 512 kbps - DSL N/A D3 64 kbps N/A 6K 128 kbps N/A 1K 256 kbps N/A 2K 512 kbps N/A 5K 1.5 Mbps $473.00 01 2.1 Mbps $623.50 02 3 Mbps $623.50 03 $882.16 04 6 Mbps $1,015.12 06 9 Mbps $1,382.63 09 10 Mbps $1,436.59 10 12 Mbps $1,467.76 12 14 Mbps $1,492.16 14 15 Mbps $1,516.92 15 18 Mbps $1,630.82 18 20 Mbps $1,688.00 20 21 Mbps $1,744.72 21 25 Mbps $1,791.60 25 27 Mbps $1,814.80 27 30 Mbps $1,849.60 30 33 Mbps $1,885.00 33 36 Mbps $2,131.43 36 40 Mbps $2,171.71 40 41 Mbps $2,183.22 41 45 Mbps $2,229.62 45 50 Mbps $2,327.06 50 65 Mbps $2,479.35 65 75 Mbps $2,660.47 75 100 Mbps $3,113.26 C2 130 Mbps $3,033.69 CX 155 Mbps $3,131.59 C3 200 Mbps $3,306.99 C4 300 Mbps $3,741.34 C6 350 Mbps $3,958.51 C7 400 Mbps $4,175.68 C8 4.5 Mbps 500 Mbps $4,610.03 CA 550 Mbps $4,810.97 CB 600 Mbps $5,011.90 CC 700 Mbps $5,413.77 CE 800 Mbps $5,783.17 CG 900 Mbps $6,152.57 CI 1000 Mbps $6,521.97 1G ¹$I= FIRN Services, RT=BANDWIDTH, NNNN=Any Numbering System, YR=Year Service Was Ordered, OC=Option Code. ²Additional 1Mbps BW applies to connections over 200 Mbps. 2.5 Optional Services 2.5.1 Extended Demarc FIRN uses the MFN flat rate pricing for the extension on the customer’s Demarc at a rate of $273.75 per circuit installation. This is the end user rate. Extension of the Demarc consists of a mounting containing a loopable device and an 8-pin jack that allows the customer to connect their CPE. In most cases, this equipment will be installed at the “minimum point.” The minimum point is usually within 5-10 feet of the customer’s entrance cable. If needed, the technician will install additional wiring and an 8-pin interface jack from the demarcation point located at the minimum point to the customer’s CPE equipment location provided it is possible to do so. The request for extending the wiring from the Demarc can be withdrawn if it is determined that extending the wiring will require extraordinary material, labor or permit costs. The Demarc service warrants installation, including parts and workmanship made to the premises telecommunications wire and jacks against defects and malfunctions for a period of one (1) year from the date of installation. The Demarc service warrants for a period of ninety (90) days all labor materials provided for restoring customer’s wiring and/or equipment to good working order.   Expedite process – FIRN will use the existing MFN expedite process. After Hours charges – FIRN will use the existing MFN After hours process 2.6 SLA Overview FIRN uses a highly available and highly reliable network backed by stringent network performance and operational service level commitments. These commitments are based upon guaranteed response times and other performance measurements, with associated user credits for service provider non-compliance. The FIRN service levels are designed to ensure required performance and delivery expectations are met. Service levels will be applicable on a per incident basis, and will apply to all FIRN provided Internet communications including the physical layer of the network. Installation SLA’s will be applicable after initial migration period. FIRN service includes pro-active Service Level Assurances (SLA’s) with automatic credits. Should a trouble condition be experienced, a trouble ticket will be proactively opened by the MFN Network Operations Center (NOC) after initial triage by Hayes. Once a trouble ticket has been issued by the NOC, users and SUNCOM NOC representatives will work together to restore service outages and/or resolve service issues. Should a SLA violation occur, appropriate credits will be applied to the impacted user’s account and will be capped at 100% of the site’s monthly billing. The SUNCOM NOC will have the ability to monitor and verify SLA adherence via the web based MFN Network Management System. DMS-SUNCOM will meet with AT&T on a monthly basis to review outage reports as part of managing these service levels. Any applicable service provider non-performance penalties will be credited to the user’s monthly invoice for the affected site. For the process to receive credits, please refer to the section SLA Credits to DMS/End User Agency and Dispute Process. 2.7 SLA Categories The Service Levels are divided into two categories as listed in the bullets below:   Outage and Service Troubles Operational Outage and Service Troubles Service levels for the FIRN Outage and Service Troubles will be measured in terms of service outage as defined in the matrix below Restoral thresholds are defined with associated user credits for service provider non-compliance. SLA New FIRN Services - Service Level Agreement Site Outages & Service Troubles - Respond Site Outages & Service Troubles - Restore Performance Target Outage & Service Troubles Liquidated Damages Within two (2) Hours 10% MRC of entire service if notification > 2 hours. Within four (4) Hours 15% MRC of the entire service if outage > 4 hours. *MRC of Service = MRC (or Monthly Recurring Service) of DAN, Bundle or Unbundled services for each site. Optional services network services are excluded from SLAs. Operational General network operations and administration service levels will be applied on a per incident basis, including and as defined in the matrix below.   Installations Move, adds and changes SLA New FIRN Services - Service Level Agreement Performance Target Operational Liquidated Damages 64kbps to T1 = 25 business days Install, Moves, Adds, Changes ("IMAC") >T1 to 45Mbps = 40 business days 25% MRC of service* if performance is not met >45Mbps = 90 business days *MRC of Service = MRC (or Monthly Recurring Service) of (Core Port + CPE + Access) for each site Service Level Component Definition:  Install, Moves, Adds, Changes (service intervals) A service interval starts when a CSAB order is accepted by AT&T and ends when the service is installed. This interval is applicable to an individual site location only. Users are responsible for providing accurate CSAB order information. Any delays caused by the user will not be counted toward these intervals. 2.8 Service Levels Exclusions Please note the following exclusions from service levels and credits:        SLAs will not be applicable when CPE is managed by the user and is determined to be the cause of the outage. FIRN service levels are excluded during scheduled maintenance windows. SLA’s are not applicable if access is not purchased through FIRN and is determined to be the cause of the outage. . If access facilities are not available for bandwidth speeds defined install intervals will be excluded. Any installation or restoration delays caused by a user will not be part of the specified installation or service restoration intervals. FIRN service levels will not be applicable due to Force Majeure. For example, delays directly due to acts of God, wars, acts of public enemies, strikes, fires, floods, or other similar cause wholly beyond the providers control, or for any of the foregoing that affect subcontractors or suppliers if no alternate source of supply is available to the provider. Desk-top support issues reported to or referred to Hayes with no SLA metric will be documented and tracked in the Hayes ticketing system Keystone 2.9 FIRN SLA Applicability – Service Types FIRN SLAs for end-users are categorized into two types as listed below. For other details on SLA and these categories, please refer to the FIRN contract and other section related to SLA in this FIRN OPS guide. 2.9.1 FIRN SLA Categories: 2.9.1.1 Outage and Service Troubles 2.9.1.2 Operational 2.9.2 FIRN DAN, Bundled and Unbundled 2.9.2.1 All SLA categories apply. See Section 10 3 FIRN Customer Support 3.1 Trouble Management Support Trouble Management Support is provided by the MFN Network Operations Center (aka FIRN NOC). The MFN NOC provides remote proactive monitoring of customer networks and systems using centralized remote monitoring tools and technical personnel. The MFN NOC is in operation 24 hours a day, 7 days a week, 365 days a year, for coordination and resolution of network events. The MFN NOC proactively monitors all aspects of the fault, configuration, accounting (network usage, user access, configuration changes, etc…), performance and security as it pertains to the FIRN network. 3.2 Trouble Reporting Procedures After a site has been provisioned on the FIRN, customers will need to follow the procedures described in this section to initiate a Trouble Ticket. Network problems are normally identified by the MFN NOC in a proactive manner but a trouble ticket can be generated by either calling the Hayes NOC at 1-877 347-6896 or *email to [email protected] or the MFN NOC directly at 1-866-913-8386 or email to [email protected] For trouble management, the Network Management Systems (NMS) will continually monitor the Network infrastructure and the customer’s CPE devices. Tier 1 technician will be responsible for the items listed in the “MFN NOC Responsibilities” in Section 3.4. The MFN NOC will immediately open a trouble ticket in any of these cases. * The MFN NOC staff will respond with a ticket number after the email has been read; normally this will occur within 30 minutes. If this is a critical situation, or if you do not receive a response within 30 minutes, please call the MFN NOC immediately @ 1-866- 913-8386. Notes At no time will a customer be instructed to “call another department or partner” to address their issue. Every effort will be made to hand the customer off directly via warm transfer to the appropriate support team member or they will be advised that the appropriate team member will contact them by a specified time. For FIRN Direct Connect** customer outages the Tier1 MFN NOC can expect to receive an informational call, from the AT&T NOC, to be advised of the outage and given an AT&T NOC trouble ticket number. The AT&T NOC will not troubleshoot the outage. MFN NOC maintains responsibility for troubleshooting the outage. Customer managed CPE has minimum requirements to allow monitoring by the NMS as specified in the NMS CPE Requirements for Monitoring. **Direct Connect refers to customers installed under Change Request 46 (CR-46). 3.3 Customer Responsibilities 3.3.1 Contact information: To maintain proper communications and system effectiveness the FIRN customer must maintain the designated contact information and keep it up to date. See table below with this information. For FIRN services at an existing location, this information can be updated by the Agency Security Administrator by calling or emailing the MFN NOC. The MFN NOC will then open an informational ticket to perform the updates. If the person requesting the change is different than the person on file, the MFN NOC will ask customer to submit a ORDER to DMSSUNCOM to authorize an update to the security administrator. For requested FIRN services at a new location, all types of contact information will be done through the CSAB order process. The MFN NOC will - add the contact information to the Remedy System. How to update or add a security administrator: 1) 2) 3) Customer submits a complete ORDER to DMS-SUNCOM with contact information including phone number and email address. DMS-SUNCOM will submit the request to AT&T to process. AT&T in turn will forward the request to MFN NOC to update the Remedy System For urgent changes, the SUNCOM NOC will be contacted at 1-888-4SUNCOM (Option 1, 9) or [email protected] to validate and authorize the contact information. Once validated, an email will be sent by the SUNCOM NOC to the MFN NOC to update the Remedy system. The SUNCOM NOC will advise the customer that this method will allow for a temporary change in the system. Within 5 business days, the customer must follow up with a ORDER to make this a permanent change. If the customer does not submit a complete ORDER within 5 business days, the contact information will revert to that from before the request was made. After 5 business days, the SUNCOM NOC will follow up with the customer to find out if a ORDER has been submitted or not. Failure to keep this information current may result in delays in restoring the service and could adversely affect the ability to meet any Service Level Agreements (SLAs). In addition, the request to access MFN Portal (NMS Tools) will be denied. Contact information required for each location: Site Hours of Operation Special instructions for NOC related to After Hours contact Contact Type Primary Technical Secondary Technical After Hours Technical Name Phone Number Email Insert Hours Available Security Administrator MFN NOC Changes Technical Gathering Primary Technical Secondary Technical After Hours Technical Security Administrator MFN NOC Changes Definition Agency OR LOCAL Site Technical contact. This contact will work with MFN NOC to resolve troubles and coordinate ACCESS to the facility when needed. Backup to Primary when Primary not available. Agency OR LOCAL Site Technical contact(s) to be contacted 'after-hours' to work with MFN NOC to resolve troubles and coordinate ACCESS to facility when needed. Please indicate HOURS available to contact. The Agency person with the authority to add or delete users’ access to the NMS Tools and or Remedy and to approve contact names and numbers. The Agency person(s) with the authority to request changes to Vendor managed CPE from the MFN NOC. Technical Gathering 3.3.2 The individual responsible for gathering the information necessary to complete this questionnaire. Submitting a trouble report Before submitting a trouble report customer should: 1) Check with their own Help Desk (if one is available) to eliminate any internal LAN issues that may be causing the trouble. 2) Consult with their designated local technical contact. 3) Check that power is available. 4) Check to determine if problem is cause by any activity recently performed on their systems (i.e., patches, maintenance, software/hardware changes, etc.) 5) Inquire internally for any other known local telecommunications outages. 6) Check the on-line Ticketing system or call the MFN NOC to verify trouble is not already reported in the system. 7) If steps 1 thru 6 above have been acted upon and there is still no troubles reported or found in the system, then proceed to report the trouble as indicated below: When calling or emailing the MFN NOC or Hayes to open a trouble ticket, please have the following information available:               Device Name (Hostname) as it appears in Spectrum CSAB Order number & circuit ID number Agency Name, Code and Location Type & description of service CPE service status (managed/unmanaged) Detailed description of the trouble Exact date and time of the trouble Contact information including name, telephone number, and access hours Site physical address Local physical site access information or contact Customer & local contact phone and cell numbers E-mail address of contact for status updates Hours of operation (When can you be contacted?) Advise MFN or Hayes NOC if you have been performing any activity on your system (i.e., patches, maintenance, software/hardware changes, etc.) or if you have any special requirements or constraints. If you submit a request to open a ticket via email and it does not contain the information referenced above, you will receive a return email requesting the information before a ticket is created and assigned to a technician. The MFN NOC will open tickets in a proactive manor based on established MFN NOC procedures as agreed to by AT&T and DMS-SUNCOM. * Note - Open trouble tickets may be viewed online by accessing the Ticketing System on the MFN NMS Web Portal at https://portal.MFN.myflorida.com. **Note: Remember that the MFN NMS system will automatically report FIRN outages and will notify the agency designated technical contact via email or SMS Paging within 15 minutes of the outage being detected as specified by the customer in the NMS User request. Reference the “Notification and Status Commitment” table in section 3.8 for details regarding notifications and status updates. 3.4 Hayes NOC Responsibilities 3.4.1 Report from customer supported by DOE or purchases the Support Option: 3.4.2 Customer contacts Hayes Help-Desk @ 1-877-347-6896. 3.4.2.1 Hayes will open a ticket in Keystone then proceed to troubleshoot the issue according to current Hayes trouble handling procedures. 3.4.2.2 In the event the issue is determined to be service affecting, related to the circuit or CPE, Hayes will contact the MFN NOC for a warm hand-off. 3.4.2.3 MFN NOC will open a ticket in Remedy then troubleshoot the issue according to current MFN trouble handling procedures. Note: For FIRN Direct Connect customers, Tier 1 MFN NOC will not have AT&T core device access to collect AT&T core connection information. 3.5 MFN NOC Responsibilities 3.5.1 Report from customer not supported by DOE and purchases options through the FIRN contract. 3.5.1.1 Customer contacts MFN NOC @ 1-866-913-8386 3.5.1.2 MFN NOC will open a ticket in Remedy then troubleshoot the issue according to current MFN trouble handling procedures. 3.5.1.3 If the MFN NOC determines during initial data gathering that the customer issue is with an option such as encryption or filtering, the MFN NOC will contact Hayes for a warm hand-off. Hayes will open a ticket in Keystone then proceed to troubleshoot the issue according to current Hayes trouble handling procedures. 3.5.2          When a customer reports a problem, the MFN NOC agrees to the following: Accept trouble reports from the customer or authorized representative by telephone or electronically (if access available). Log and track all troubles reported Test all services/facilities as necessary to resolve the problem Provide the customer with problem status periodically, as defined in “Notification and Status Commitment” table in section 3.8. Escalate troubles to higher-level support upon the customer’s request per “Escalation Procedures” defined in section 3.9. Proactively escalate ticket as necessary to Service Manager, Tier 2 and Tier 3 per MFN NOC operating procedures. Cooperatively test with the customer or authorized representative when necessary Close all tickets with the agreement of the customer or authorized representative and document the name of the individual with whom the ticket was closed Open all initial trouble tickets Provide single point of contact for communications with the customer     3.5.3 Update and monitor ticket status Forward tickets to appropriate group Check for OOB access (if applicable) Check for active alarm in Spectrum. DAN Service DAN Service Description: The DAN is a unique network to each individual school district with the host device collocated with the school district’s FIRN connected device. See Section 2.4 for description detail and diagram 3.5.3.1 Report from a District Area Network (DAN) customer: 3.5.3.2 DAN IT Staff contacts MFN NOC @ 1-866-913-8386 3.5.3.3 2MFN NOC will open a ticket in Remedy then troubleshoot the issue according to current MFN trouble handling procedures. 3.5.4 DAN Trouble Reporting requirements: 3.5.4.1 All reporting of issues affecting DAN sites will be the responsibility of the District IT staff. The District IT Staff is the single point of contact (SPOC) for all remote sites on the DAN. 3.5.4.2 Trouble report status updates and closeout information will be provided to the DAN IT staff (SPOC). 3.5.4.3 If a report comes into the MFN NOC from a location other than the DAN SPOC a ticket will be opened, the customer will be given the ticket number and advised of the process of using the DAN SPOC to report troubles. The DAN SPOC will be contacted by the MFN NOC with the ticket information and the local customer information so that the DAN SPOC can follow-up with the local customer for status and process training. 3.6 Trouble Ticket Status Check The status of the trouble ticket may be checked by reviewing the ticket in the Ticketing System through the MFN NMS Web Portal or by calling the MFN NOC at 1-866-913-8386. The address for the MFN NMS Web Portal is https://portal.MFN.myflorida.com. When calling, the customer will have available the trouble ticket number to inquire about the ticket status. Status reports will be provided to the customer by the MFN NOC staff per the “Notification and Status Commitment” table in section 3.10. Note: It is the customer’s responsibility to keep the contact information updated. For details refer to section 3.3, Customer Responsibilities. 3.7 Trouble Tickets Classifications 3.7.1 Problem Severity Classifications There are five severity classifications that a Tier 1 MFN NOC technician can assign to a detected or reported trouble. The classification assignment will be made based on the definitions of the classifications in this section.  Critical *  Major *  Minor **  Chronic ***  Informational * For all troubles, resolution efforts occur on a 24x7 basis, and status updates are provided to the customer according to the “Notification and Status Commitment” table in section 3.10, until the problem is resolved and service has been restored. Critical, Major and Minor issues have SLA’s associated. ** For ‘Minor’ troubles, resolution efforts occur primarily during regular business hours with coordinated afterhours testing with the customer to minimize interference with performance or downtime for the customer during regular business hours. There are no SLA’s associated with Minor issues unless they are change requests which carry a 2 hour SLA. Please refer to Generating Configuration Change Request section for details. *** Chronic tickets will be opened under the Major classification and noted in the problem description area as Chronic. **** Spectrum Alarm classifications do not necessarily correspond to Remedy ticket classifications. Spectrum Alarm classifications listed below: • • • Critical alarms include device outages, SLA critical thresholds exceeded, and SLA violations Major alarms include selected syslogs and traps, and device component failures (power supplies, fans, etc.). Minor alarms include non-major syslogs and traps, authentication failures, and informational alarms. Critical Critical problems are defined as those affecting the entire network for a specific agency or within the MFN core that impacts a large number of users with no immediate work around. The condition includes a critical work stoppage or service degradation that prohibits access to mission critical applications during the customer’s normal working hours affecting multiple sites within the agency. A critical condition within the MFN core would consist of a hardware or software failure that causes work stoppage or service degradation prohibiting access to mission critical applications by the connected users in a particular LATA or across the MFN core.     Examples of critical problems All network alarms for any MFN Core Router All network alarms for any MFN core aggregate circuit Multiple sites of a specific agency are down or have lost connectivity as reported by the customer or the NMS system Multiple sites are experiencing service degradation that has rendered their connections unusable as reported by the customer or the NMS system Major Major problems are defined as single site outages, service degradation prohibiting access to or function of critical applications from a single site or exceeding contracted performance SLA.    Examples of major problems Single site outages as reported by the customer or the NMS system Service degradation over a Sites WAN connection as reported by the customer or the NMS system High Priority Agency in list below (these will be given priority) Minor Minor problems are defined as affecting individual sites, and do not interrupt service, degrade performance or exceed SLA specifications to an extent that prohibits users from accessing or using critical applications. Example of minor problems Non-service affecting as reported by the customer or the NMS system Hardware performance thresholds exceeded (e.g. CPU, memory, or buffer) Latency, Jitter and Packet loss below specified parameters (SLA Table) as reported by the customer. Circuit over utilization as reported by the customer.     Informational Informational tickets are created by the MFN NOC when a customer calls to report an issue that may trigger an alarm for the MFN NOC or to request informational assistance. Informational problems do not require repair. Informational tickets regarding questions on operation of MFN tools should be submitted via email to [email protected]. Informational tickets referring to the NMS will be responded to within 72 hours. Example of informational problems  Customer reports the network will be down for maintenance  Customer reports a scheduled power outage  Customer reports equipment shutdown for office remodeling  Customer request information or clarification on MFN tools or operation Chronic A chronic ticket will be opened at the onset of the third occurrence of the same trouble type for a specific site within a 30 day period. An agency chronic ticket will be opened when an excessive number of tickets for a particular agency have been created within a 30 day period. Tickets opened under the following classifications will be excluded from the chronic ticket formula.        Customer Maintenance Customer Education Customer Equipment Duplicate Ticket Weather related UPS issue Site Power The Chronic ticket type should only be used to consolidate and track repair events within the individual outage tickets. 3.8 Notification Commitments Notification and Status Commitment Table Severity Level of the ticket Notification* Time Commitment 15 min Initial contact within 15 min of outage. Within 2 hours customer will be contacted with cause of outage and every 2 hours with status updates. Initial contact within 15 min of outage. Within 2 hours customer will be contacted with cause of outage or issue. Initial contact within 30 min of trouble and updates when conditions change. Within 2 hours customer will be contacted with cause of issue. Customer will be advised of chronic status and updated as conditions change NOC will respond to information requests within 72 hours otherwise NOC notification is not required. Critical Major 15 minutes Minor 30 minutes Chronic As Appropriate Informational As Appropriate *Notification or Status can be provided via Email** or phone within the given timeframe. See section 3.9 for “Escalation Procedures” if the stated Notification Time has been exceeded. ** There must be a valid email distribution list on file and configured for each level of ticket notification. A customer may call the MFN NOC or Log in to the Ticketing System at any time to obtain current status of a ticket. 3.9 Escalation Procedures 3.9.1 Customer Requested Escalation Procedures Escalation Step 1 At any point a customer may call to request the classification of the ticket be raised to the next higher level by calling the MFN NOC or the Hayes NOC as appropriate. For example a customer may request a MAJOR to be raised to a CRITICAL. Escalation Step 2 The customer may escalate a ticket through the MFN NOC by calling 1-866-913-8386 and asking for the manager on duty. Escalation Step 3 Customers have the option to escalate network trouble issues by contacting the SUNCOM Network Operations Center at 1-888-4SUNCOM (Option 1, 9) or email [email protected] 24 hours a day, 7 days a week. When calling the SUNCOM NOC, please refer to the procedures below:       State that you would like to escalate a MFN NOC trouble ticket Provide the MFN NOC trouble ticket number A SUNCOM NOC trouble ticket will be opened to track the progress of the escalation. The SUNCOM NOC Technician will work with the Customer to resolve the issue escalating to the SUNCOM Product Support Manager or SUNCOM Engineering Manager, as required. Further escalations within the MFN Management Tier will also be made, if requested. Once a resolution has been reached on the trouble ticket, the SUNCOM NOC Technician will verify the customer’s satisfaction with the resolution and close the ticket. 3.9.2 DMS Requested Escalation Procedures Escalation Step 1 At any point a DMS representative may call to request the classification of the ticket be raised to the next higher level by calling the MFN NOC. For Example a DMS representative may request a MAJOR to be raised to a CRITICAL. Escalation Step 2 The DMS representative may escalate a ticket through the MFN NOC by calling 1-866-913-8386 and asking for the manager on duty. Escalation Step 3 If DMS deems that further escalation is required; please refer to the table below: Level of Escalation Contact Point Contact Information Job Description 1 MyFloridaNet NOC Manager Sharon Quintero. [email protected] 850-325-3860 The MyFloridaNet NOC Manager is responsible for the daily operation network ops issues including installations and post installation troubles. 2 Director, Data Network Operations Centers Hank Adams, [email protected] 919-554-5180 The Director, Data Network Operations Centers is responsible for directing the efforts of the Tallahassee Data Engineering team as well as the Tallahassee Network Operations Center (NOC) management and work force. 3 Service Executive, Service Management Rick Chaffin [email protected] (386) 736-6351 The Service Executive handles technical and service issues that are not resolved by the normal repair process or that require special arrangements to ensure that customer needs are satisfied. 4 Director IT, GLOBAL MANAGED SERVICES Stanford L Curtiss III (810) 424-6900 office (810) 424-6900 cell Overall responsibility for IBM Bluesky accounts, AT&T Managed Services, Service Assurance, and Custom Managed Services. Level of Escalation Contact Point Contact Information Job Description 5 AVP Global Operations Matthew Lucas (949) 838-8183 office (949) 887-3525 cell Responsible for Tier 1.5 through Tier 3 support of IBM Bluesky, AT&T Managed Services, Outsourcing, Custom Managed Services and MSS customers via work centers based in the US, Bangalore, India and Campinas, Brazil. 6 VP Global Enterprise Managed Services John Walsh (919) 474-1200 office The Global Enterprise Managed Services (GEMS) organization is responsible for domestic and global service assurance through Global Customer Service Centers (GCSC’s) supporting MRS; eVPN; MIS; and AVTS Service Lines. Additionally, the team has responsibility for Global Technology & Tier 3 Services, Global Managed Voice Services, Managed Security Services and Globally Managed Mainframe Services. 7 Senior VP Service Assurance Eric Boyer (214) 757-3080 office The Global Enterprise Managed Services (GEMS) organization is responsible for domestic and global service assurance through Global Customer Service Centers (GCSC’s) supporting MRS; eVPN; MIS; and AVTS Service Lines. Additionally, the team has responsibility for Global Technology & Tier 3 Services, Global Managed Voice Services, Managed Security Services and Globally Managed Mainframe Services. Level of Escalation Corporate Sponsor Contact Point EVP Global Customer Service Contact Information Andrew Geisse (214) 757-5820 Job Description Chief Technology Officer for BellSouth Corporation. In this role, he is responsible for setting the technology direction of BellSouth's core infrastructure. His department includes, network and operations technology, Internet protocol (IP) applications, next generation strategy as well as BellSouth Entertainment, LLC. . 3.10 Generating Configuration Change Request Most changes to the FIRN services including all billable items will require the customer to submit a ORDER through the Online CSAB system. Please refer to the chapter 4 in this guide “Ordering FIRN Services”. There are certain non-billable item changes that do not require a ORDER. For details please refer to the following tables. Changes identified in the FIRN NOC column marked “Yes” can be accomplished with a NOC ticket. FIRN Managed FIRN Elements and Features FIRN Service Content Filtering Hourly maintenance Monthly maintenance Expedite After Hour Installation Special Construction Extended Demarc LAN IP addressing maintained by DOE LAN IP addressing provided by Agency Internet Encryption - All types Multicast FIRN Managed CPE Access List Firewall Feature Set QoS type - voice, video, application NAT IP Helper Address add / change DHCP LAN interface settings (Speed, Duplex) CSAB Order Required Establish New Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes No Yes No Change Existing ** Yes Yes Yes Yes N/A N/A N/A Yes Yes No Yes Yes No Yes No No No No No No No NOC Ticket Establish New No No No No No No No No No Yes/Hayes No No No No No No No No Yes No Yes Change Existing ** No No No No No No No No No Yes/Hayes No No Yes No Yes Yes Yes Yes Yes Yes Yes IP Accounting (Troubleshooting Aid) Route Cache Flow (Troubleshooting Aid) Add Static Route Elements and Features not listed above No No No Yes No No No Yes No No Yes No Yes Yes Yes No 3.11 Customer Managed Option Customer Managed FIRN Elements and Features FIRN Service Content Filtering Hourly maintenance Monthly maintenance Expedite After Hour Installation Special Construction Extended Demarc LAN IP addressing maintained by DMS / DOE LAN IP addressing provided by Agency Internet Encryption – All types Multicast Customer Managed CPE Access List Firewall Feature Set QoS type – voice, video, application NAT IP Helper Address add / change DHCP Add Static Route Elements and Features not listed above CSAB Order Required NOC Ticket Establish New Change Existing Establish New Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes ** No Yes No No No No Yes Yes Yes Yes Yes N/A N/A N/A Yes Yes No Yes No No Yes ** No No No No No No Yes No No No No No No No No No Yes/Hayes No No No No ** No No No No No Yes No Change Existing No No No No No No No No No Yes/Hayes No Yes Yes No ** No Yes No No No Yes No ** CSAB Orders can be used for any changes the customer would like to request. The advantage to a CSAB Order is engineering review for these changes. NOC tickets will reduce implementation time. **** All required information must be in the ticket to complete the requested changes. If the information is incomplete the ticket and the changes will be placed on hold until all required information has been provided by the requestor. For any Customer requested changes that may have an effect on the MFN core, the MFN Engineering and Design team will meet to review and discuss the design change before any changes can be made. For any changes requiring a ORDER or for changes not listed in the previous tables, Customers should initiate and submit the ORDER via the online CSAB system. Please see chapter 4 for more information on how to order changes. Based on the tables above, when a change is requested through the MFN NOC, the user should provide the information listed below. For any changes requiring a ORDER, please refer to Chapter 4 of this guide.            Circuit ID Number Type of service Managed or unmanaged CPE Detailed description of the request Exact date and time needed for the change CSAB Order Number Customer contact name Customer phone and cell numbers Email address of contact for status updates Hours of operation (When can you be contacted?) Any special requirements or constraints. The status of a request can be checked by telephoning or emailing the MFN NOC (1-866-913-8386 or [email protected]) or by using the Ticketing System application in the MFN Web Portal at http://portal.mfn.myflorida.com. For tickets initiated by Hayes the user should contact Hayes at 1-877 347-6896 or email Hayes at [email protected] . Router configuration changes that require engineering review and/or testing may take longer. 3.12 Hours of Operation The MFN NOC is operational 24 hours a day, 7 days a week, 365 days a year via live assistance. 3.13 MFN NOC Contacts, Escalation Levels & Responsibilities 3.13.1 Contacts and escalation levels for MFN Operations are as follows: CenturyLink Service Type Contact/Escalation Level Level 1 Phone Number MFN NOC 1-866 -913-8386 [email protected] Call 866-913-8386 ask for a supervisor Level 2 FIRN Sharon Quintero – NOC Supervisor 850-325-3860 ofc [email protected] Level 3 Hank Adams CenturyLink Data Service Director [email protected] 919-554-5180 ofc AT&T Service Type Contact/Escalation Level Level 2 Phone Number AT&T Distribution (includes all contacts listed below) [email protected] Rick Chaffin – Service Manager 386-736-6351 [email protected] FIRN Cassius George Provisioning Liaison 404-499-5458 [email protected] Level 3 Alex Oliver – NOC Manager [email protected] 404-499-5388 Hayes Service Type Contact/Escalation Level Phone Number Level 1 K-12 – Hayes 1-877 347-6896 ask for Duty Supervisor [email protected] 2 Lee Vickery ANS Engineer [email protected] 850-297-0551 X 166 FIRN Level 3 John Strobel – Director of Special Projects [email protected] 850-297-0551 X 115 Tier 4 – Design Engineering Service Type Contact/Escalation Level Level 1 Core Phone Number Mark Sullivan Lead Core Engineer 850-216-3552 [email protected] Level 1 CPE Gardner Smith Design Engineer 404-499-5521 [email protected] FIRN Level 2 Dale King Lead Design Engineer 404-499-5522 [email protected] Level 3 Lalitha Parameswaran Supervisor Engineering 831-594-3041 [email protected] DMS-SUNCOM Engineering and Operations Service Type Contact/Escalation Level Phone Number FIRN SUNCOM NOC 850-413-9569 4 Ordering FIRN Services 4.1 Customer Engagement The following methods can be used to discuss and order FIRN Services.         AT&T Account Executives contact the customer to propose services AT&T Account Executives discover a Customer need during routine meetings with customer. The Customer contacts the AT&T Account Executive with a need, DMS-SUNCOM personnel contact the customer to propose services DMS-SUNCOM personnel discover a Customer need during routine meetings with customer. The Customer contacts DMS-SUNCOM personnel with a need. The Customer places an order for services on the SUNCOM Website. The Customer contacts Department of Education for assistance or for DOE If the AT&T Account Team is going out to meet with a Customer in reference to FIRN Services, they will notify the corresponding local AT&T Account Manager who will notify the DMS-SUNCOM Sales consultant or notify the DMS-SUNCOM Sales consultant directly as appropriate for that customer as listed in the tables below. Additionally, AT&T will keep DMS-SUNCOM Sales Consultants informed with all communications and correspondence with the customer pertaining to FIRN. For example, AT&T emailing the customer will CC the consultants within that region. When meeting with SUNCOM customers or potential customers, AT&T will present Suncom as the network of choice and primary service offering. 4.1.1 Customer Engagement Personnel Consultant Telephone Email Counties Denise Adkins John Bellows 850-921-1647 [email protected] (850) 9227486 or 1-8884SUNCOM (1-888-4786266) Option 4 [email protected] DMS Consultants Manager Bay Calhoun Escambia Franklin Gadsden Holmes Gulf Jackson Liberty Okaloosa Santa Rosa Walton Washington Jefferson Leon Madison Taylor Wakulla Marvin Powell (850) 4137906 1-8884SUNCOM (1-888-4786266) Option 4 [email protected] Alachua Baker Bradford Clay Columbia Dixie Duval Hamilton Union Gilchrist Lafayette Levy Marion Nassau Putnam St. Johns Suwannee revised 8/03 Janet Doherty 941-3737599 or 18884SUNCOM (1-888-4786266) Option 4 [email protected] Brevard Flagler Indian River Volusia Orange Okeechobee Bill Fox (352) 3301363 or 1-8884SUNCOM (1-888-4786266) Option 4 [email protected] Charlotte Citrus DeSoto Hardee Hernando Highlands Lake Manatee Hillsborough Pasco Pinellas Polk Sarasota Sumter Linda Myers (305) 4705098 or 1-8884SUNCOM (1-888-4786266) Option 4 [email protected] Broward Collier Glades Hendry Lee Martin Palm, Dade, Monroe Service Delivery Telephone Email Services Kevin Langston Jennifer Swanson 850-922-7477 [email protected] Chief Service Delivery 850-921-4067 [email protected] Derek Howard Steve Welsh 850-922-7474 [email protected] 850-414-7235 [email protected] Joelle Peek 850-410-0009 [email protected] Toll Free, VPN, Reservationless, Switched PIC, and air card. FIRN , LD Dedicated (by agency) FIRN, LD Dedicated (by agency) Assist on all services when assigned. AT&T State Government Account Managers Account Manager Telephone Email Areas Danny Thomas 850-216-3553 [email protected] State Account Manager Supervisor Rob Gass 407-826-6707 [email protected] Chuck Lang 850-216-3535 [email protected] Kevin Patten 850-617-1862 [email protected] State Government Account Manager State Government Account Manager State Government Account Manager AT&T Local Government Account Managers Account Manager Telephone Email Areas Jeff Parsons 904-359-7211 [email protected] Tom Henderson 850-969-7002 [email protected] Chris Wadley 904-359-7248 [email protected] Daphne Dilbert 813-878-3276 [email protected] Ismael Gonzalez 407-245-2106 [email protected] Thomas Gill (Sr) 407-245-2135 [email protected] Wiley Horton 352-371-5572 [email protected] Account Manager Telephone Email Jacksonville Area Local Government Account manager Supervisor Pensacola Local Government Account Manager Jacksonville Area Local Government Account Manager Tampa Area Local Government Account Manager Daytona Beach Area Local Government Account Manager Orlando Area Local Government Account Manager Gainesville Area Local Government Account Manager Areas Esperanza Diaz-Bello 305-569-7236 [email protected] Elena Cordal 305-569-7292 [email protected] Ester Martin 305-569-7274 [email protected] Margarita Castellon 954-838-1721 [email protected] Miami Local Government Account Manager Supervisor Miami Area Local Government Account Manager Miami Area Local Government Account Manager Fort Lauderdale Area Local Government Account Manager Maria Johnston 305-582-9475 [email protected] Miami Area Local Government Account Manager Bill Daniel 561-640-6630 [email protected] West Palm Area Local Government Account Manager Nancy Vinez 561-640-6668 [email protected] West Palm/Ft Pierce Area Local Government Account Manager Miriam Buonomo 305-840-2389 [email protected] Miami Area Local Government Account Manager 4.2 Prerequisites for Ordering During the initial stages of the ordering process, the FIRN Team involved with the Customer may consist in all or part of the following personnel:       DMS-SUNCOM Sales Consultants DMS-SUNCOM Service Delivery Personnel AT&T Account Executives AT&T Technical Consultant AT&T Solutions Sales Engineer Customer personnel 4.2.1 Documents and Resources Documents in Appendix 15.5: Resources:      Information on the SUNCOM Website FIRN Network Pricing - refer to SUNCOM Website Expedite Process (See Section 2.5 - Optional Services) After Hours ((See Section 2.5 - Optional Services) User Guide 4.2.2 Speed Selection Customers will choose connection speed from the FIRN offering. The Access method and Router selection will be made by the vendor. It is understood that higher bandwidth and / or additional requirements or services may require an enhanced router be deployed. 4.2.3 QoS QoS deployment will require engineering prior to submitting the ORDER. A QoS Template should be completed and attached to the ORDER. DMS will work with the customer and AT&T to complete the template For a copy of the QoS template, go to Appendix 14.12. 4.2.4 Service Inquiry If the customer has need for high-speed connections, above 9 Meg or Ethernet, the team will need to submit a Service Inquiry request to AT&T to determine the availability of service for the location(s) in question. Service Inquiries will require the following information to be submitted:      Speed of connection, If upgrade, speed of current connection Valid site address, Local contact name and number, Telephone number for the location if different from the local contact’s number. In cases were telephone service does not exist for a location, a working number near the location can be used and should be reported as such. Service Inquiries normally take up to 7 business days to determine service availability. Results of the Service Inquiries will be incorporated into the CSAB order by the customer or DMS. . 4.2.5 Special Construction Occasionally it may be determined that while facilities were not available for installation of service as determined in the FIRN Service Inquiry, facilities can be constructed to provide the requested service at the desired location. This will require additional time and the customer may incur additional expenses. The requirements, expenses or any additional time required for special construction will be presented to the customer and approval obtained prior to submitting the ORDER to DMS to order services. If facilities are not available AT&T (and its subcontractors) special construction charges will be approved by DMS Service delivery with customer written authorization. Approval will be expressly stated on the ORDER and the cost of Special Construction included as part of the overall cost for the service on the ORDER ordering document submitted by the customer for ordering FIRN service for the subject location. 4.2.6 Site Readiness For any non-Fiber based FIRN services, the site receiving FIRN service must be ready to receive the service and have a space appropriate for housing network equipment. This includes the site being fully constructed and ready for occupancy and having a secure, clean, environmentally conditioned, properly lit space with the necessary electrical power and with equipment racks, floor space and/or plywood backboards available. All of the following requirements must be met before the circuit installation can be completed. Is there an AC power outlet available? (Yes or No) Is there adequate lighting in the room? (Yes or No) Is there sufficient backboard space and a #6 ground wire? (Yes or No) Is wiring/infrastructure already in place (inside the building) from your Demarc to your final router location? (Yes or No) If No, when will it be in place? Special Access required for room? (Yes or No) Is there rack or shelf space available? (Yes or No) How and where should the network equipment be placed / mounted? (Please describe: shelf, wall, rack?) Is conduit needed from the street to the building? (Yes or No). If Yes, when will it be in place? Does the equipment room have temperature and humidity control? (Yes or No) List any special room conditions. List none if there no special room conditions. Has the Certificate of Occupancy been issued to customer? (Yes or No) Note: Any site readiness requirements will be at customer expense. For any fiber based FIRN services, a site survey will be performed to determine if the site is ready for fiber based FIRN services. This process will take place after DMS submits a complete ORDER to the service provider. A service provider representative will contact the customer to schedule a site survey. 4.3 Initial Service CSAB Order Submission – Installs Orders will be reviewed by SUNCOM Service Delivery personnel before submission to AT&T. After an acceptable design has been established, it is SUNCOM’s goal to review and submit ORDER’s to AT&T from the customer within three business days of receipt. Additionally, ORDER’s will be correct and with an acceptable, functional design at submission. Any assistance required by DMS-SUNCOM from AT&T will be sought out and obtained prior to ORDER submittal. DMS-SUNCOM will engage DMS-SUNCOM Engineering, AT&T Technical Consultants and/or AT&T Solutions Sales Engineers, for assistance before submission of the ORDER to AT&T. AT&T will assist DMS with the design and review within 3 business days. 4.4 In-House Relocations In-house relocation of D-Marcs will be initiated by ORDER. In-house relocations can typically be completed much quicker than new circuit installations and will not be subject to the standard intervals for installation. AT&T will work with SUNCOM to perform in-house moves in a quick and timely manner. 4.5 Relocation of Service to New Physical Address Relocation of service to a different physical address will be initiated by a CSAB order and will typically be handled by issuing two (2) ORDER’s. A separate ORDER will be issued for installation of the new service at the new location and a second ORDER will be issued for the disconnection of the existing service. This is necessary as customers often want no down time associated with the move and require service continuation at the original site until the relocation of all personnel and equipment is completed. If downtime can be tolerated by the customer and the customer is not requesting an upgrade of the existing service, the router may be re-used/relocated to the new service location. This will require approval and coordination with the customer by DMS-SUNCOM or AT&T. Rented routers should be moved to the new location by AT&T or one of its subcontractors and not the customer. 4.6 Changes to Existing Service Most changes to the FIRN services including all billable items will require the customer to submit a ORDER through the Online CSAB system. There are certain non-billable item changes that do not require an ORDER. For details please refer to the following tables. Changes identified in the FIRN NOC column marked “Yes” can be accomplished with a NOC ticket. ***Changes that can be requested by an FIRN NOC ticket will be completed within 2 hours of creation of the ticket. The ticket will be created with a severity classification of Minor. 4.7 FIRN Managed Option FIRN Managed FIRN Elements and Features FIRN Service Content Filtering Hourly maintenance Monthly maintenance Expedite After Hour Installation Special Construction Extended Demarc LAN IP addressing maintained by DOE LAN IP addressing provided by Agency Internet Encryption - All types Multicast FIRN Managed CPE Access List Firewall Feature Set QoS type - voice, video, application NAT IP Helper Address add / change DHCP LAN interface settings (Speed, Duplex) IP Accounting (Troubleshooting Aid) Route Cache Flow (Troubleshooting Aid) Add Static Route Elements and Features not listed above CSAB Order Required Establish New Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes No Yes No No No No Yes Change Existing ** Yes Yes Yes Yes N/A N/A N/A Yes Yes No Yes Yes No Yes No No No No No No No No No No Yes NOC Ticket Establish New No No No No No No No No No Yes/Hayes No No No No No No No No Yes No Yes No No Yes No Change Existing ** No No No No No No No No No Yes/Hayes No No Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No 4.8 Customer Managed Option Customer Managed FIRN Elements and Features CSAB Order Required Establish New Change Existing NOC Ticket Establish New Change Existing FIRN Service Yes Yes No No Content Filtering Hourly maintenance Monthly maintenance Expedite After Hour Installation Special Construction Extended Demarc LAN IP addressing maintained by DMS / DOE LAN IP addressing provided by Agency Internet Encryption – All types Multicast Customer Managed CPE Access List Firewall Feature Set QoS type – voice, video, application NAT IP Helper Address add / change DHCP Add Static Route Elements and Features not listed above Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes *** No Yes No No No No Yes Yes Yes Yes N/A N/A N/A Yes Yes No Yes No No Yes *** No No No No No No Yes No No No No No No No No Yes/Hayes No No No No *** No No No No No Yes No No No No No No No No No Yes/Hayes No Yes Yes No *** No Yes No No No Yes No ** ORDERs can be used for any changes the customer would like to request. The advantage to a ORDER is engineering review for these changes. NOC tickets will reduce implementation time. * All Private IP Addresses must be validated with the MFN NOC prior to implementation to ensure no conflict will arise with any other IP Addressing Scheme in production *** An ORDER will be required if a customer is requesting access to resources outside their VRF or to another Agency. Site to Site ACL creation or changes within an Agency do not require a ORDER. DMZ site between multiple sites within a particular Agency, with pre-defined layer 4 ports does not require ORDER. The 2 hour configuration SLA does not apply to the creation of any new Access List. 6-23-10 note – while it is ideal to make some changes using a NOC ticket, a DMS directive has been issued to require a CSA for any ACL changes until further notice. **** All required information must be in the ticket to complete the requested changes. If the information is incomplete the ticket and the changes will be placed on hold until all required information has been provided by the requestor. 4.9 Changes Requiring a CSAB Number Change Changes in Speed will result in a CSAB Authorization Number change and will be processed as a disconnect of existing service and installation of new service. Two ORDERs will be issued to assist in the changing of AT&T invoicing and DMS-SUNCOM inventory and billing records. The ORDERs will clearly state the reason for the change and that a physical disconnect and installation of the service may not be required unless a new local loop is needed. 4.10 Discontinuation of Service Discontinuation of service will be initiated by DMS-SUNCOM through the issuance of a CSAB order. The ORDER will be issued with the same SUNCOM Inventory ID number as the installing ORDER and will contain at a minimum:     agency location name, address, city, phone number local contact and phone number, requested due date for discontinuation of service Circuit number. AT&T will honor the customer requested due date (CRDD) when it meets or is beyond the standard disconnect interval of 10 calendar days after receipt of the ORDER from DMS to AT&T. DMS may request shorter intervals on an exception basis. AT&T will consider granting shorter intervals on a case by case basis. 4.11 Mandatory Use of Online CSAB System All ORDERs will be submitted to AT&T by DMS-SUNCOM and accepted by AT&T via the Online CSAB System or similar system as deployed by DMS-SUNCOM. Closeout information required by DMS-SUNCOM will be provided via the Online CSAB system prior to payment or acknowledgment of installation, change, or service discontinuation. Changes approved via NOC ticket and not affecting invoicing charges would be an exception. 4.12 Incomplete CSAB Orders* ORDERs submitted to AT&T will contain all the necessary information as agreed to by DMS-SUNCOM and AT&T. Any ORDER that is incomplete or missing information should be brought immediately to the attention of the DMS-SUNCOM personnel issuing the ORDER for correction. This can be accomplished in real-time if all parties are available or may be returned to DMS-SUNCOM for resolution if DMS-SUNCOM personnel are unavailable. For details and process on SLA hold time, refer to section 4.13.4. DMS-SUNCOM will contact the customer for clarification to complete the ORDER and return to AT&T for processing. Any modification to the requirements set forth by the ORDER must be approved by DMS-SUNCOM before processing or payment will not be authorized. *No ONS services can be ordered without selecting an access service or connection first. 4.13 FIRN Installation Process 1. A call will be made by the Vendor to the on-site contact prior to the due date to advise when to expect the installation of the circuit to occur. 2. Circuit technicians will be dispatched to install the new FIRN circuit and the Out of Band Modem line (in AT&T / Legacy BellSouth Territory) prior to installation of the router. 3. A call will be made to the on-site contact prior to the due date to advise when to expect the installation of the router to occur. 4. The CPE installer will be dispatched on the planned Test and Turn-Up date as close to the beginning of the requested appointment window as possible. 5. The CPE installer will work with the Provisioning Personnel to turn up the equipment on the circuit and verify connectivity to the Core and to the internet. 6. The CPE Installer will work with the customer in connecting the router to the Customer’s LAN if the customer is ready at the time of turn up. 7. If there are any unresolved issues noted during the testing process, the customer will be left on their existing service and a new installation date will be rescheduled. 8. After the successful testing and turn up of the customer location the provisioning Engineer will close out the Remedy provisioning schema. After closing of the Remedy Provisioning schema an email will be sent to the NMS team to populate the tools. 9. The systems team will work to populate the device in to the Network Management System (NMS) Tools. The sites are usually populated within 2 business days. 10. Based on the guidelines stated in this document related to service turn-up and effective bill dates, AT&T and it partners will enter the “Completion Date” and “Effective Bill Date” in the Online CSAB system. AT&T and its partners will close ORDERs in the online CSAB system within 7 business days from the day the service was turned-up. DMS will not authorize payment if ORDER’s are not closed out properly in the Online CSAB System. It is extremely important that AT&T and its partners close out ORDERs in a timely manner following this rule. 11. The following table shows those individuals ATT has authorized to work and close FIRN ORDERs in the Online CSAB system. This list will be maintained by ATT and any changes will be reported monthly. While this table is intended to show the individuals who will be processing ORDERs for FIRN it is understood that in emergencies ATT may have to add resources in order to meet an emergency need of the State. It should be noted that while the individuals below are listed as those who are currently approved and tasked to work FIRN ORDERs, any ORDER activity performed by any employee of ATT, any Subcontractor acting on behalf of ATT, or any other individual given access to the Online CSAB System by ATT, ATT has ultimate responsibility for all activity associated with the ATT Online CSAB account. Name Flo Brekka Beth Daly Phone Number 954-838-1538 954-838-1582 Email Address [email protected] [email protected] Partner AT&T AT&T Name Maura Cordero Phone Number Cathy Bevis Jon Osterbur John McCoy 954 838-1431 407-628-6649 850-847-0199 Email Address Partner [email protected] 954 838-1339 AT&T [email protected] [email protected] [email protected] AT&T Centurylink Centurylink 12. Billing will commence the day following successful turn-up and testing/cutover to FIRN. For example: Please refer to section 5.2 “Billing Start Date for new services” for additional information. 4.14 Installs, Moves, Adds, and Changes (IMAC) SLAs 4.14.1 SLA Start and End Requirements 4.14.1.1 An SLA performance target begins when the following items are complete: 1. a complete and accurate CSAB order is submitted by the customer to DMS 2. it is reviewed by DMS for accuracy and completeness 3. is submitted to the Service Provider 4.14.1.2 The SLA performance target ends: For Vendor Provided CPE / Managed: 1. The router has been installed on the customer premise 2. Router has been turned up with the Provisioning engineer and engineer has documented activities in the provisioning comments section of Remedy 3. Customer verifies and accepts installation by signing the FIRN Installation Checklist as a template for services to be tested. Testing may be performed using the Installers Laptop per customer request. Note: The customer may or may not cutover their LAN at this point based on customer request. For Customer Provided CPE / Unmanaged: 1. The Vendor will contact the customer to advise of the circuit installation due date. This is a minimum of two days prior to circuit completion. 2. The circuit will be installed and tested by Vendor Network technicians prior to the turn up of the CPE. 3. The Vendor will contact the customer to set an appointment for turn up of the CPE. 4. The Vendor will send a confirmation email of the appointment time for the turn up of the CPE, which will include expectations on billing start date to the customer, The Vendor will have configured the CORE Interface prior to the appointment time. 5. The turn up appointment will be scheduled by the Vendor and could take place within 7 business days of the circuit install date or billing will commence automatically. Refer to section 5.2.2 (Customer Provided CPE) for details. 6. The customer will configure their CPE using the guidelines and site specific technical data provided by the Vendor to them for the location. 7. The customer will connect the CPE to the access circuit. 8. The customer will be responsible to call the Vendor at the scheduled appointment time to work with the Provisioning Engineer to confirm circuit operation and routing through the CORE. 9. The customer will be responsible for any LAN side connections of the CPE and any cutover activities that need to take place for new service turn up. 10. If the customer does not contact the Vendor on the scheduled appointment time, the Vendor will close out the turn up with the scheduled date and the Technical contact as being completed. 4.15 SLA Installation Performance Targets The table below provides the SLA installation performance targets for the FIRN by bandwidth speeds. The SLA report will represent business days as method for calculation. SLA Targets exclude the Vendor holidays SLA Installation Performance Target Bandwidth Range SLA Target* All FIRN Except Secure Internet Services Install within 45 business days (Approximately 35 calendar days) 64kbps to T1 = 60 days Secure Internet Services >T1 to 45Mbps = 80 business days >45Mbps = 180 business days If the SLA performance target is not met, AT&T will credit DMS 25%, 10% for Secure Internet Services, of their customer service Monthly Recurring Charge (MRC). SLA violation credits are not applicable on DMS cost recovery. A customer should request a due date that is not less than the above SLA performance targets. It is the customer’s responsibility to plan and forecast for services ahead of time. This will allow DMS ample time to work with the customer on ORDER requirements such as design review (if necessary), service inquiry, and validation of other requirements for a submittal of a complete ORDER. Refer to the Chapter 4.0 on Order Process for additional information. Incomplete information on the customer ORDER will impact the due date and SLA performance targets. The customer may request a shorter due date as a Best Effort from Vendor, in which case AT&T will determine if the request can be met without expedites. SLA Installation performance Targets will still apply if the dates cannot be improved. The customer may also request a shorter interval by using the Expedite procedure as detailed in Section 2.4.2 . In the case of an expedite request the Vendor will make every effort to meet the requested expedite date. SLA Installation Performance Targets are not applicable to expedited order time improvements. DMS will work with AT&T to determine if the requested due date can be met and notify the customer with any issues. 4.16 Other Services and impact on SLA Performance Target The following services can be completed much quicker than a new circuit installation and will not be subject to the standard intervals for installation as described above. The vendor will work with SUNCOM and make every effort to complete these quickly and in a timely manner. Requested Service Change CPE upgrade on existing circuit CPE module upgrade on existing CPE and Circuit Expectation* 15 15 In house relocations Extension Demarc on existing service Optional Features such QoS, Multicast, Encryption on existing circuit using the CSAB process Bandwidth upgrade on existing circuit. For example, if a customer has a 12 Mbps circuit provisioned on a DS3 and the customer want to upgrade to a DS3. 15 15 15 15 *Business Days 4.17 CSAB order submittal and SLA Performance Target for new installations Below is a summary of what is included and excluded from the SLA performance target for some of the major requirements of a CSAB ORDER. As compared to other CSAB order requirements, these have a larger impact on dues dates and SLA performance targets. (Before or After) CSAB order Submittal to AT&T (included or excluded) SLA Performance Target Before excluded After included Before Before and / or After excluded Site Survey (fiber based) After included Site Readiness (fiber based) After excluded CSAB order requirements Service Inquiry - above 12 Mbps or all Ethernet Service Inquiry – 12 Mbps and below Customer Service Questionnaire Review by DMS Site Readiness Requirements (non-fiber based) excluded Example: A “Service Inquiry – above 12 Mbps or all Ethernet” as a CSAB ORDER requirement must be performed “Before” the CSAB ORDER is submitted to AT&T. The time it takes to perform this CSAB ORDER requirement is “excluded” from the SLA performance target. Note: For any item that is excluded from SLA performance targets after a ORDER has been submitted to AT&T, an SLA Hold time needs to be documented in the Remedy Order for the location as noted in the process below. 4.18 Book of Record and SLA Hold Time Guidelines The Online CSAB system will be the book of Record for order issuance to the Vendor and completion dates from the Vendor. The Remedy System will have an order issued for each new installation and will track any “SLA Hold Times” against the SLA Performance Targets. The Remedy Order Start Date will be entered using the date to the ORDER was issued to AT&T based on the Online CSAB System. The Remedy Order End Date will be populated with the same date that will be entered in the Online CSAB System as the “Completion Date”. The Completion Date is based on the day of Service Turn-Up for Vendor Managed or the Appointment time set for Customer Managed. For SLA Hold Time only, the Remedy System will be used for the IMAC data on the SLA monthly Report. Any “SLA Hold Times” must contain the three details listed below to be considered for review. Discretion will be used when using “SLA Hold Times” reason. 1. 2. 3. Detail Reason of SLA Hold Time Timestamp for start of SLA Hold Time Timestamp for end of SLA Hold Time Any SLA Hold Time with the above information will be validated as part of the SLA scrubbing process by the SUNCOM NOC and AT&T (and it sub-contractors). The SUNCOM NOC will validate and compare all the SLA reporting information with the information provided through the Online CSAB system. Any SLA Hold Time without the above information (three bullet items) will not qualify for SLA Hold Times. 4.19 SLA Hold Time Request and Approval Process The vendor will email DMS at [email protected] and [email protected] to request a SLA hold on a ORDER. The email should provide as much detail as possible as to the reason of the SLA hold time. This will be followed up by a phone call to the DMS Service Delivery personnel (listed on the ORDER) as well. DMS Service Delivery personnel will respond in a like manner (email & call) to notify the vendor if the request is approved or rejected. If approved, the vendor will return the ORDER via the On-line CSAB System to DMS adding in the comment the SLA Hold Reason and Hold Times (start and end). See a list of valid SLA hold reasons below in section 4.13.4.2 “SLA Hold Time Reasons”. DMS Service Delivery personnel will accept or reject the Hold Times by adding a comment in the Online CSAB System. In addition, the vendor will update the Remedy Order System with this information The SLA clock will be stopped while DMS Service Delivery corrects the ORDER and will be restarted once the ORDER is received by AT&T for processing. However any delay in determining that a ORDER is incomplete does not relieve AT&T of its SLA intervals for installation of service. In other words, a ORDER cannot be held for a length of time and then returned to DMS Service Delivery for correction and have the clock restarted on the SLA process. The SLA clock is only suspended while the ORDER is at DMS Service Delivery for change or completion. Every effort will be made by both the vendor and DMS Service Delivery personnel to contact the customer and fix the issue as soon as possible in order to remove the SLA Hold time. If rejected and disputed, DMS service delivery will escalate the issue to their management for resolution. No ORDER will be returned or put on hold without prior approval from DMS Service Delivery personnel. 4.20 SLA Hold Time Reasons Category CSA CSA CSA CSA CSA Hold reason Incorrect Address Customer Unavailable Unresponsive to calls Unresponsive to emails Unaware of Order Design Design Design Design Other Insufficient QoS parameters Special Routing Requirements Inappropriate CPE Configuration Site Readiness Site Readiness Site Readiness Site Readiness Certificate of Occupancy Ground/backboard (Copper) Conduit (Copper) Fiber-based requirements not met by agreed upon date Test & Turn Up Test & Turn Up Test & Turn Up Customer Not ready Customer changes/Moves the Due Date Technical person not Ready or unavailable 4.20.1 Other Impacts on SLA Performance Targets   If access facilities are not available for bandwidth speeds greater than 12 Mbps and all Ethernet, defined SLA performance targets will not be applicable Installation delays caused by end user owned equipment 5 Billing 5.1 Billing Cycles Services are initiated through the DMS On-Line CSAB (Communications Services Authorization) process in the Order Tracking System. AT&T will complete Block 7 –“For Telco Use Only” by providing the Work Order Number, Completion Date, Circuit number and Effective Bill Date as well as other required fields on the ORDER. The first chargeable day will be the day as stated on the ORDER Effective Bill Date. This effective bill date will be based on the guidelines described in this document including section 5.2 below and “FIRN Installation Process”. It is the responsibility of SUNCOM Billing to validate the invoices against the “Effective Bill Date” for accuracy. Additionally, SUNCOM billing will validate that billing has started the next calendar day after the completion date. See example below. Any discrepancies will be reported to AT&T. Fractional charges, which are a portion of the Monthly Recurring Charges (MRC), will be billed from the Effective Bill Date up to the next month’s bill cycle. For example: Completion Date: Effective Bill Date: Initial Bill Date: Fractional Charges: MRC: 10/12/2009 10/13/2009 11/01/2009 10/13/2009 – 10/31/2009 11/01/2009-11/30/2009 5.2 Minimum Billing Periods There is no minimum billing period in the FIRN contract. Customers will be billed for the service they have in order to comply with the Erate requirement that the service provided must be the service billed. 5.3 Billing Start Date for new services Billing for installation of new services on FIRN falls in to two categories based on CPE acquisition as defined in sections 7.4 (Acquired) and 7.4 (Customer Provided CPE). The differences are detailed below: 5.4 CPE acquired through the FIRN contract All new FIRN equipment will be staged, configured and installed as part of the packaged CPE pricing. If the FIRN turned up on Friday, billing will start on the following calendar day as the effective bill date. 5.5 Customer Provided CPE Customers choosing to use / provide their own CPE acquired from other sources are an integral part of the turn up process on FIRN. The Vendor will work with the customer to schedule turn up appointments within 7 business days of the access circuit installation. The turn up schedule will be mutually agreed upon by the Vendor and the Customer and will be used as the record for setting billing dates. The FIRN billing will start the next calendar day after the scheduled CPE provisioning date for the site. For example, if the Access circuit was installed on Monday, the CPE was turned up on Friday then the billing will start on the following calendar day as the effective bill date. If there is any delay in completing the Turn-up due to a Vendor error, billing will be deferred to start the next calendar day after the correction of the error. 5.6 Disconnect Processing and Effective Bill Date For disconnection of service, AT&T will provide the same information in Block 7 and the Effective Bill Date as agreed upon. Standard disconnect interval is 10 calendar days after receipt of the ORDER from DMS to AT&T. DMS may request shorter intervals on an exception basis. AT&T will consider granting shorter intervals on a case by case basis. For example: Complete disconnects ORDER Received: Completion Date: Effective Bill Date: Bill Date: Fraction Credit: 10/15/2009 10/25//2009 10/26/2009 11/01/2009 10/26/2009 – 10/31/2009 Any move/change that is downgrading, upgrading or site change etc. that is not completely disconnecting service and replacing existing one will not be double billed. For example, billing for old/existing service would stop the day of the completed move and the effective bill date of the new install will be the next calendar day. Please note both a Disconnect and New Install ORDER must be issued. For example: Change of service: New Service turned up: Existing ORDER Bills through: New Service starts Billing: 10/15/2009 10/15/2009 10/16/2009 AT&T is required to provide all information for Block 7 – “For Telco Use Only” as soon as the orders are completed. This process affects the billing to customers whether installing, cancelling or making changes to existing services. For installs and disconnects, DMS-SUNCOM must validate the effective date used within the customer billing and this information must be completed by AT&T within the current bill cycle if at all possible. DMS-SUNCOM will use the validated date provided unless the customer challenges the date; if this occurs a formal dispute will be initiated. See process in section 5.11 below. 5.7 Change in billing results from a funding change Proviso language by the Legislature, or a responsibility change DMS will submit a ORDER to AT&T to effect this change. For changes reflecting new FRN numbers ORDERs should be issued no later than May 1st prior to the next funding year. Requests to change billing responsibility will be initiated by DMS through the issuance of a ORDER and will typically be handled by issuing two (2) ORDER’s. A ORDER will be issued under the agency code of the new agency for the installed service at the existing location and a second ORDER will be issued for the cancellation of billing for the existing service. There are at least three situations where this billing change may apply: 1. The original Agency passes “Billing only” responsibility to the new Agency All provisioning and support responsibility remains with the original Agency. In this event, the Remedy ID and NMS Tools access remains with the original Agency. AT&T will create a “Billing only” record in ccConnect to effect the change in billing. The ORDER reflected in the billing to DMS will be changed to reflect the New ORDER. Close-out comments on the ORDER’s will reflect a “Billing only” change. To properly manage the potential discontinuance of service, it is prudent on DMS and AT&T to maintain a cross-reference of both ORDERs. Record and will process this change request through the downstream systems to affect the following at a minimum: o No changes required in the downstream provisioning and support systems o Remedy ID o Router Name, as required o Router Description to include the new ORDER number Example: DOE is no longer responsible for billing under Agency code for the individual FIRN users; however, DOE maintains full responsibility for the installed service under Agency code 48R. 2. The original Agency passes all billing, provisioning and support to the new agency. In this event, there is no change required to the provisioned service. AT&T will create a new ccConnect Record and will process this change request through the downstream systems to affect the following at a minimum: o Remedy ID o Router Name, as required o Router Description to include the new ORDER number o NMS Tools Security Administrator, if required Example: The DOE will no longer fund the Community Colleges’ Internet Access. All services currently installed under DOE Agency code 48S will be move to the appropriate Community College code i.e. C14 for Manatee Community College. Some planning was done during the migration in preparation of this potential change in funding such as allowing all Community Colleges access to their devices within NMS Tools. 3. The original Agency passes all billing, provisioning and support to the new agency. In this event, there is a change required to the provisioned service. AT&T will create a new ccConnect Record and will process this change request through all downstream systems to affect the following at a minimum: o New Agency Name o New Contact Information o New Router configuration including a change in IP Address and VRF Assignment o Remedy ID o Router Name o Router Description to include the new ORDER number o NMS Tools Security Administrator In this situation time is required to process and schedule the provisioning of the service to the new Agency. Example: Due to proviso language, DOE will pass ownership another agency or District. The physical location will not change, however all provisioning will be changed to reflect specific details of the new agency or District. This change will follow the process of a move of service without changing the physical address. The ORDER will be issued with the new Authorization number and will contain at a minimum: • The new ORDER Authorization Number • Agency location name, address, city, phone number • Local contact and phone number, • Requested due date for change of service • Circuit number • VRF assignment • IP Address changes • Explanation of the type of Billing Change being requested. AT&T is required to provide all information for Block 7 – “For Telco Use Only” as soon as the orders are completed. This process affects the billing to customers whether installing, cancelling or making changes to existing services. For installs and disconnects, DMS must validate the effective date used within the customer billing and this information must be completed by AT&T in a timely manner. DMS- will use the date provided unless the customer challenges the date; if this occurs a formal dispute will be initiated; see process in section 5.11 below. 5.8 Credits and Dispute Resolution Guidelines All charges billed by AT&T to DMS- DIVTEL must be authorized by a ORDER. If charges are billed that are not authorized by a ORDER – they will be deducted. DMS- DIVTEL will not pay AT&T for unauthorized charges and will deduct the unauthorized charges from the monthly bill with notification by email to AT&T that the charges are not valid. When AT&T issues the credit for the unauthorized charges; the credit will not be accepted; the charges were not paid, therefore the credit is not due to DMS- DIVTEL. AT&T issued the credit to offset charges previously billed. This process is to clear the past due amount from the billing. For other disputes within the billing; DMS- DIVTEL will pay the charges with an agreed upon time period for AT&T to correct the billing. The normal time period for corrections will be 2 bill cycles. For example: Dispute identified on June 5th bill cycle: Email to AT&T identifying dispute: Correction received by bill cycle: 6/5/2009 7/2/2009 9/5/2009 Some corrections may occur sooner depending on the date the dispute is identified. Based on this example, if the dispute was identified to AT&T on 6/15/2008 then the correction should be made by the 8/5/2008 bill cycle. If the correction is not received within two billing cycles, the charges will be deducted from the next bill. If this occurs, email will be sent to notify AT&T of the deduction. 6 NMS Tools 6.1 Network Management Systems CPE Requirements for Monitoring Customers managing their own CPE and desiring to have their CPE monitored by the NMS tools must comply with the following configuration requirements in order to be monitored:  Read Only SNMP security string that can be utilized by the MFN network management servers. This information must be shared with the MFN network management provisioning team prior to routers being added to the systems.  Expanded access control list (ACL) to allow the devices to be reached by all necessary FIRN IP network management server addresses. This list of around 40 IP addresses includes CA Spectrum and CA eHealth servers located in Tallahassee, FL (primary location), and Winter Park, FL (failover location). Cisco Example: remark EMBARQ-Spectrum-tlh-1 permit x.x.x.x  Configure SNMP trap destinations. This will forward SNMP trap data to the Spectrum server, which is necessary for fully functional fault management. Cisco Example: snmp-server host x.x.x.x vrrp frame-relay envmon flash authenticatefail bgp sdllc config-copy config entity event-manager hsrp ipmulticast mvpn ospf pim syslog aaa_server atm firewall ipsec isakmp rtr snmp  Configure a syslog destination to include the MFN syslog server. functionality of configuration management and security software. This is required for complete Cisco Example: logging trap notifications logging x.x.x.x  Follow the FIRN naming convention (FRN+LATA+City Code+Agency Code+incremental number) in order for the tools to function and provide secure limiting of customer views. Customers not able to rename devices will still see the FIRN names utilized in the tools. Note: All required configuration details and parameters will be supplied to the end user by AT&T during planning meetings before turn up. 6.2 MFN Network Management Tools User Training Users are training on the MFN Network Management tools through three approaches. 1. Online web based training 2. Local or web based instructor led training 3. Vendor classes  Online On Demand Web Based Training Tool overview and navigational training is available online from the DMS FIRN Training Website and updated regularly.  Local or web based instructor led training Special in depth classes are held at DMS facilities or at agency locations in Tallahassee. If users are remote, web based live training is supplied if possible. Training agendas are customized according to customer needs. Classes are being led by experts involved in installing and maintaining the systems. Contact: [email protected]  Vendor Classes Classes for CA eHealth and NetQoS Reporter Analyzer are occasionally held by software vendors at DMS training facilities. These special classes are scheduled by the Embarq sales team. Contact: [email protected] 6.3 MFN NMS Tools Access Requests 6.3.1 Additions – The process below is not active yet. Currently, we are using the NMS Tools form to approve access to the Web Portal. All (including new and existing Customer, DMS and vendor employees) NMS Tools request will be sent to [email protected] for DMS approvals. Refer to the User Guide process for additional information. The Agency ORDER Administrator should complete and submit the ORDER through the online CSAB system. Requestors should assure that all information on the ORDER is filled out in order to avoid delays in processing. Please view the sample ORDER in Appendix 14.11. DMS will submit the ORDER via the Web-based CSAB system to AT&T. AT&T will forward the ORDER to the Embarq - Network Systems inbox for completion of the request. The Embarq - Network Systems Group will e-mail the user notifying them that the MFN NMS Tools account has been configured and that the Embarq – Network Systems Group will contact the user to provide account and password information. The Embarq notification form is located in Appendix 14.11. Note that the Q-Radar Security tool is limited to two accounts per organization. It is advised that the Q-Radar user have an IT Security Background. Only an organization Security Administrator may request and authorize an account. DMS will handle the NMS Tools accounts for Customers and AT&T employees (and its partners) through the CSAB online system. NMS Tools accounts or any other MFN access for AT&T and its Partner employees will only be created after the statutorily required Background Check is satisfactorily completed (see MFN OPS Guide Chapter 11 “Background Checks for MFN”). Customer Access to MFN web portal (NMS tools) will be denied if the customer security administrator contact cannot be validated. 6.4 Routine Changes The Agency ORDER Administrator should complete and submit the ORDER through the online CSAB system. Please view the sample ORDER in Appendix 14.11 6.5 Routine Removals If an existing working account is not needed, it is the responsibility of the agency ORDER administrator to submit a ORDER to delete the account. Requests to delete accounts should be sent from the agency through the online CSAB system. Please view the sample ORDER in Appendix 14.11. DMS will submit the ORDER via the Web-based CSAB system to AT&T. AT&T will forward the ORDER to the Embarq - Network Systems inbox for completion of the request and the ORDER can then be closed. 6.6 Emergency Account Removals In the case of an emergency account removal request, DMS should forward the information to [email protected], and then escalate the request by calling the MFN NOC. The MFN NOC will contact the appropriate manager to expedite the removal, including after hours and weekends. 6.7 Network Management Reports Network management reports are available to end users and DMS from various products and can be accessed from the web based user interface. Section 6.5 provides information on 1. CA Spectrum Report Manager 2. CA eHealth 3. NetQoS Reporter Analyzer CA Spectrum Report Manager Reports Report Types CA Spectrum Report Manager (SRM) is used by Embarq Network Operations and SLA analyst teams to generate reports for performance SLA alarms from Spectrum to be included in monthly SLA Report. Spectrum SPM SLA Report The Report Manager also provides useful inventory reports. Spectrum SPM Asset Report CA eHealth Reports CA eHealth is used by DMS, and internal organizations to report historical data on devices and connections for customer sites and the core network. Report Types  At-a-Glance The following eHealth reports are available from the console and the Web interface. They provide a series of charts that show the performance of critical variables for a specified element during the report period. Reports are typically generated for Router/Switches, LAN/WAN connections, and QoS. CA eHealth At-A-Glance Report  Trend Reports The following reports are available from the console or the Web interface. They show the performance of an element or a group of elements, over a specified period of time, based on specific variables. Reports are typically generated for Router/Switches, LAN/WAN connections, and QoS. CA eHealth Trend Report Scheduling Reports Users can generate the “At-a-Glance” or “Trend” reports on demand. If the user has a need to create a custom report, a request needs to be submitted to your local SUNCOM Sales Consultant 1-888-4SUNCOM (Option 4). A meeting will be scheduled with the customer to discuss the request. DMS will work with the AT&T and Embarq on the report. As an example, if a user would like a trend report on a connection to include data for one full year, this would not be feasible from the user interface and would take an excessive amount of time to run. In this scenario the user would run a similar report for a short time period, and then make the request through the support address to have this report scheduled for after hours with an extended time period. Reports such as this can take hours to run from the interface, and run just a few minutes after hours as a background process. NetQoS Reporter Analyzer Reports Traffic analysis from NetQoS Reporter Analyzer is done from a web based interface, typically on a real time basis to troubleshoot issues or analyze network traffic patterns. Report Types  Real Time Reports Real time reports include Protocol, ToS, Flows and Utilization for the previous two hour period with 1 minute granularity. This reporting is ideal for real time troubleshooting of issues. NetQoS Reporter Analyzer Real Time Report  Historical Reports Historical reports include Protocol, ToS, Flows and Utilization for selectable time periods for up to 1 year with 15 minute granularity. This type of reporting is ideal for looking at previous time periods, and provides more reporting options and flexibility. NetQoS Reporter Analyzer Historical Report Reports Access and Scheduling Information Available from Web Interface? CA Spectrum Report Manager SLA Reports Inventory Reports Manual generation of reports by end user? Automatic generation of reports by end user? Reports available online? Reports available via e-mail? yes yes yes yes yes yes yes yes yes yes yes yes yes yes no* no* yes yes yes yes Real Time Traffic Analysis Reports yes yes no yes no Historical Traffic Analysis Reports yes yes no yes no CA eHealth At-A-Glance Reports Trend Reports Netqos Reporter Analyzer * Can be scheduled by system administrators. Recommended if reports are for extended time periods or required on a recurring basis. 7 Customer Premise Equipment (CPE) 7.1 FIRN CPE CPE will be included in the cost for the packages under the new FIRN contract. Customer Provided CPE Option Customers may use existing routers on the FIRN network. To ensure proper operation with the FIRN network and to qualify for applicable SLAs, the customer must load these routers with the current approved FIRN IOS and must be router models currently supported by the manufacturer. Special needs will be handled on a case by case basis by DMS. 7.2 Maintenance Maintenance supporting the standard FIRN restoral SLAs is included in the rental prices of FIRN CPE Customers acquiring new equipment from sources other than the FIRN contract (after 4/9/2007) will not be eligible for FIRN CPE maintenance or associated SLAs and will also be responsible for CPE trouble shooting and coordinating any necessary CPE repairs themselves. Provided the customer follows FIRN configuration guidelines, FIRN CPE management is available and the MFN NOC will at the customer’s direction monitor, backup configurations and notify the customer when CPE or other outages are detected even if the CPE is not under FIRN maintenance. FIRN installation services are not available for CPE purchased outside the FIRN contract. Key features of CPE maintenance include:  Service restoral with 24 x 7 x 2,4 and 8-hour basis  24x7x365 access to MFN NOC, engineering resources and CPE vendor Technical Assistance Centers  IOS software updates  24x7x365 access to CPE vendor websites such as Cisco Connection Online (CCO) Other maintenance needs should be addressed to the Customer’s DMS representative. 7.3 Installation Installation for CPE on FIRN falls into two categories as defined in sections 7.1 (FIRN CPE Acquisition Options) and 7.2 (Customer Provided CPE). The differences are detailed below: 7.4 Acquired All new FIRN equipment will be staged, configured and installed as part of the package pricing. Billing will start the day after a successful CPE installation. Customer Provided CPE: Customers choosing to use / provide their own CPE obtained from other sources will be given: the minimum hardware recommendations per the approved CPE Roadmap for the connection speed, the proper configuration guidelines and site specific technical data / information. The customer will be responsible for configuring the CPE for a given site’s installation. The service and customer provided CPE installations for these sites will be coordinated with the customer by the Vendor Implementer. The Turn-up schedule will be mutually agreed upon by the Customer and the Vendor, documentation will be sent via email with the agreed upon Turn-up schedule and will be used as the record for setting billing dates. The FIRN billing will start the day after the scheduled CPE Turn-up date for the site. The core and access circuits may actually be installed several days in advance of the scheduled Turn-up date but that will have no impact on billing. Billing will start per the schedule unless installation delays are caused by the FIRN Vendor. Turn-up is expected to be scheduled and completed within 7 business days of circuit installation. The turn up appointment will be scheduled by the Vendor and take place within 7 business days of the circuit install date or billing will commence automatically. Refer to section 5.2.2 for details. Installation procedures and expectations are documented in section 4.12. If there is any delay in completing the Turn-up due to a Vendor error, billing will be deferred to start the day after the correction of the error. The preferred strategy is to have all Customer locations installed as Managed sites. 7.5 Out of Band (OOB) Access For managed CPE, the FIRN vendor in AT&T territory will install (at no cost) OOB access (dial up line and modem) at T1 and greater sites. This OOB access is a crucial aid in troubleshooting, allowing the verification of site power, the retrieval of router and WAN interface diagnostic information and the quick restoral of the configuration when it is necessary to replace the router. In spite of the $0 cost and the many advantages of OOB from a maintenance and management perspective, it is understood some customers may choose to not allow OOB access. This choice will affect customer SLAs as noted in section 7.10 below. 7.6 Router Configuration Backup The MFN NMS tools will maintain the last 25 copies of each routers stored configuration along with the userID of who made each change as long as the MFN NOC has read SNMP access. The MFN NMS includes a configuration management tool that facilitates storage and gives the FIRN user many abilities including the side by side comparison of configurations. Unmanaged customers must follow proper configuration guidelines that enable configuration management tool access to their routers. Unless other customer specific solutions are able to be arranged through DMS, unmanaged customers are responsible for configuring replacement routers after maintenance events and will be able to access the stored configurations via the MFN Portal. 7.7 Router Management Full Management Full router management is included and available as an option for all supported CPE at no additional charge and includes CPE configuration management, CPE MIB Polling, syslog trap support, NMS tools access, performance reports, proactive trouble responses, enhanced security support and CPE user access Management via TACAS and/or the CPE and Core proxy. If full management is selected as an option, customer’s CPE will be managed by the MFN NOC and customers will have read-only access to their routers via the MFN Portal. Router configuration changes can be made using a MFN NOC ticket or issuing an ORDER. (See table in section 4.8) Those changes that require engineering review and/or testing may take longer. Read/Write access combined with OOB to the CPE allows the MFN NOC to deliver the following capabilities:      IOS Upgrades (where appropriate – rental, leased or maintained CPE) Configuration Management of the Router 2,4, and 8 Hour Restoral Performance Degradation Isolation CPE, Circuit or Core Trouble Isolation Unbundled Customer Managed 1 - Customers may opt to provide and manage their own CPE routers. FIRN does not support simultaneous Read/Write capability for the MFN NOC and the customer. Customers allowing the MFN NOC Read Only access combined with OOB to the CPE receive the following reduced capabilities from the MFN NOC:   4 Hour Service Restoral After Joint Trouble Isolation Has Occurred Limited CPE, Circuit or Core Trouble Isolation and Resolution 2 - Self managed customers allowing the MFN NOC Read Only access combined with no OOB to the CPE receive the following reduced capabilities from the MFN NOC:   4 Hour CPE and/or Circuit Restoral After Customer Led Trouble Isolation Has Occurred Circuit or Core Trouble Resolution 3 - Self managed customers not allowing the MFN NOC Read Only access combined with no OOB to the CPE receive the following greatly reduced capabilities from the MFN NOC:   4 Hour Circuit Restoral After Customer Led Trouble Isolation Has Occurred Circuit or Core Trouble Resolution 7.8 Router Access Account management The method of controlling access to FIRN routers is through the FIRN provided integrated LDAP and TACAS+ server at no additional charge. New accounts will normally be activated within 72 hours. Customers choosing selfmanagement will be responsible for their own router access methods and account management. It is highly recommended for trouble shooting and maintenance purposes that self-managed Customers allow the MFNFIRN NOC read/only access to customer managed CPE routers. 7.9 After Hours Maintenance Activity When troubles occur and an on-site visit to the customer premises is required either to replace CPE or for circuit maintenance, the Customer must provide a live on-site contact that will be at the customer premises to receive replacement CPE and/or to allow the FIRN technician access to the site. Any delays associated with the FIRN NOC being able to contact/verify an on-site contact will correspondingly lengthen the FIRN SLAs. It is particularly important that customers designate and keep up to date, appropriate on-site contacts for sites that desire after hours restoral of service. Regardless of the success or failure of contacting an on-site customer representative, the FIRN NOC will troubleshoot, particularly circuit issues, to the fullest extent possible. 7.10 FIRN CPE SLA Matrix The following matrix is provided as a reference guide to illustrate the impact of the information provided in the rest of this document. It also provides additional detail regarding the impact of customer selecting to take advantage of the various FIRN CPE management, OOB and other options and the effects of Customer selections on FIRN management, maintenance and SLA’s. CPE Access & Features 4 Hour CPE Restoral ** 4 Hour Access Restoral** CPE SLAs (Jitter, Latency, Packet Loss) Break, Fix, IOS Upgrades* Router Configuration Backup NMS Tools FIRN NOC 2 Hour Router Changes Access Authentication (TACACS+server) Customer Managed Customer CPE – FIRN CPE – Customer Customer Managed Managed Vendor Managed FIRN CPE – FIRN Customer CPE – FIRN Managed Managed With FIRN Maintenance * Without FIRN Maintenance With FIRN Maintenance Without FIRN Maintenance With FIRN Maintenance Without FIRN Maintenance With FIRN Maintenance YES YES NO YES YES YES NO YES YES YES NO YES YES YES NO YES YES YES YES YES YES YES YES YES YES YES YES NO YES YES YES YES YES NO YES YES YES YES YES NO YES YES YES YES YES NO YES YES NO NO NO NO YES YES YES YES NO NO NO NO YES YES YES YES Without FIRN Maintenance Customer Access (Read/Write RW RW RW RW RO RO RO RO Capability) FIRN NOC Access (Read/Write RO RO RO RO RW RW RW RW Capability) Proactive Monitoring YES YES YES YES YES YES YES YES * FIRN maintenance only available for supported CPE procured the FIRN contract or that existed prior to 04/09/2007. ** SLA clock starts immediately for customers allowing OOB. For non-OOB, SLA clock starts after power verification and joint problem determination by AT&T and the Customer. For customer managed sites, circuit and CPE Restoral clock starts after joint determination has been made by the Customer and the FIRN NOC on which item to dispatch. 7.11 FIRN CPE Roadmap / Models A copy of a most recent FIRN CPE roadmap can found on eWorkspace. 7.12 General The CPE Roadmap is a list of accepted CPE for use on FIRN. These models underwent strenuous acceptance testing for approved operation with the Core equipment on the FIRN Network. The list of tested models will be posted eWorkspace. This web site link lists all of the Standard models and does not include specially approved options or models for specific applications. These specially approved models will be listed under the USOC listing or CPE spreadsheet. A copy of these most recent documents can be obtained through AT&T. 7.13 Special CPE needs Periodically customers will have special requirements for CPE. The following steps cover the addition of CPE / Features / Options for these situations: 1. During the Discovery / Design meetings with an Agency a need for a particular option or feature is discovered. 2. The AT&T and DMS FIRN Engineering team is engaged to review the requirement. 3. DMS will be provided with the test plans and if requested by DMS to participate in these tests. 4. The Hardware / Software / Feature(s) are scheduled to be tested in the Lab. This testing will commence as soon as it can be scheduled. No specific timeframe can be assigned to this task as the constraints are variable. The testing constraints include but are not limited to: Hardware availability, IOS availability, Lab Availability and or Customer availability. The length of time required to complete testing will vary based on complexity of the requirements and resources required to complete testing. 5. DMS will have remote access to these labs to conduct and observe desired test scenarios in real-time. 6. Upon test completion, DMS will be provided with the test results. 7. Upon successful completion of the Lab Testing, the Sales Engineering Team will be notified to publish pricing and request USOCS. 8. USOCS will be requested for new items, this may take 60 Calendar days to complete. 9. AT&T will notify DMS of new CPE, USOC and pricing. This will include the configuration of such CPE 7.14 New CPE models Periodically new models of CPE will need to be added over the life of the contract. This may be due to new features and option only available on new models or normal replacement of model series by the manufacturer. The following steps cover the addition of new CPE models: 1. DMS, Customer or AT&T can request the addition of a new CPE model. This will be done through a Change request. 2. AT&T will submit the Change Request (CR) to DMS describing the changes or additions. This CR will be utilized to place all requirements including features/options, hardware/software configuration, test plans and results, pricing for the approval process. 3. The AT&T and DMS FIRN Engineering team is engaged to review the requirements through this CR. 4. DMS will be provided with the test plans and if requested by DMS to participate in these tests. 5. The new CPE model Hardware / Software / Feature(s) are scheduled to be tested in the Lab. This testing will commence as soon as it can be scheduled. No specific timeframe can be assigned to this task as the constraints are variable. The testing constraints include but are not limited to: Hardware availability, IOS availability, Lab Availability and or Customer availability. The length of time required to complete testing will vary based on complexity of the requirements and resources required to complete testing. 6. DMS will have remote access to these labs to conduct and observe desired test scenarios in real-time. 7. Upon test completion, DMS will be provided with the test results and placed into the CR. 8. Upon successful completion of the Lab Testing, the Sales Engineering Team will be notified to publish pricing and request USOCS. 9. USOCS will be requested for new items, this may take 60 Calendar days to complete. 10. AT&T will notify DMS of new feature, USOC and pricing. This will include the configuration of such CPE. This information will be placed in the CR and other applicable documents such CPE road map and CPE pricing spreadsheet. 7.15 FIRN CPE Naming Convention Methodology When naming FIRN CPE, the following naming conventions must be followed. 1. The router model number 2. A dash 3. One of the following:   On standard routers, the transport the router is configured for, such as T1, 2T1, ME, DSL For one-off, custom configurations which are not expected to be generally used, "XX" will be used instead of the transport. 4. A dash 5. One of the following:   Standard Routers: a feature set designation, such as SEC or HSEC An agency identifier, such as HSMV, FIN, or JAC 6. A dash 7. One of the following:   Other special modification/additions to the router, such as 4FE for 4 switched 10/100 ports. One off: a sequence number, 01, 02, etc. as the agency generates new custom configurations. 8. If additional qualifying information is needed it will be added to the end of the router name and separated with an additional dash. Example of CPE Names: CPE Type Cisco 1841 (BGP/HSEC) Cisco 2821 Custom 7206 for HSMV Cisco 1841 (BGP/Security) CPE Name 1841-ME-SEC 2811-4T1-SEC 7206-XX-HSMV-01 1841-64K-SEC 8 Network Engineering & Design 8.1 Design Overview This chapter describes the basic MyFloridaNet design and provides a description of the requirements for the VPN routing and forwarding (VRF) connectivity as well as the network access and traffic routing requirements and considerations. MyFloridaNet IP Routed Core is a design based on Layer 3 VPNs using general specifications contained in RFC 4364. RFC 4364 VPNs are also known as BGP/MPLS VPNs because BGP is used to distribute VPN routing information across the MyFloridaNet backbone and MPLS is used to forward VPN traffic across the backbone to remote VPN sites. Customer networks, because they are private, can use either public addresses or private addresses, as defined in RFC 1918, Address Allocation for Private Internets. When customer networks that use private addresses connect to the public Internet infrastructure, the private addresses might overlap with the same private addresses used by other network users. MPLS/BGP VPNs solve this problem by adding a VPN identifier prefix to each address from a particular VPN site, thereby creating an address that is unique both within the VPN and within the public Internet. In addition, each VPN has its own VPN-specific routing table that contains the routing information for that VPN only. 8.1.1 VPN Routing and Forwarding Tables (VRFs) To separate a VPN's routes from routes in the public Internet or those in other VPNs, the PE router creates a separate routing table for each VPN, called a VPN routing and forwarding (VRF) table. The PE router creates one VRF table for each VPN that has a connection to a CE router. Any customer or site that belongs to the VPN can access only the routes in the VRF tables for that VPN. 8.2 VRF Design Internet CPE Public VRF Agency VRF Behind Agency FW Only CPE CPE Agency Firewall CPE Agency CPE Centralized Firewalls CPE Common Services VRF Agency CPE Stand Alone Agency VRF Agency CPE CPE CPE Agency HQ CPE CPE Agency VRF Behind Agency FW and State FW CPE Agency Firewall CPE FIRN VRF The FIRN VRF denotes a Layer 3 VPN on the MyFloridaNet backbone that is not firewalled. As the term Public signifies, this routing & forwarding space should be considered the same as the open Internet and is therefore unsecured.    All connections to the FIRN VRF must rely on their own local Firewalls and security measures. No route-target leaking will be done into or out of the FIRN VRF. For a customer in the FIRN VRF, State Network access would be via a separate extranet connection. 8.2 8.2 Routing Protocol Requirements CE-PE routing methods supported will vary according to the type VRF the CE is connecting to, with a preferred routing method for each scenario. Table 1 summarizes the supported methods with the preferred method in bold. Table of Supported CE-PE Routing Methods VRF FIRN Static yes BGP YES Notes:     The choice of routing protocol will have a significant impact on CE-PE convergence times. BGP provides better convergence times as compared to other routing protocols and hence it is recommended to deploy BGP. Connections to the FIRN VRF shall have inbound route-filters applied that will only accept routes from the CE that have been agreed upon with the customer. An inbound packet-filter will also be applied that will only allow in packets with source-addresses that are from the same agreed upon subnet list. Static routes will be the preferred method for LAN connectivity on managed CPE routers. 8.3 IP QoS Requirements The table below provides the IP QoS requirements for each class of application. Class Description DSCP Marking Voice Voice over IP EF 46 Video Interactive Video AF41 34 Application Priority Data AF21 18 Best Effort All other Traffic (Internet, ftp) BE 0 Signaling Emergency Voice Call setup & control AF31/CS6 26/48 Priority VOIP AF43 38 DSCP (Decimal Value) Notes:      IP QoS is available for the following access types - Frame Relay, ATM, Private Line (Dedicated) and Metro Ethernet. Minimum Bandwidth for QoS is 512 Kbps. 512 Kbps circuits can allocate 250 Kbps of voice or 250 Kbps of a data application. 768 Kbps circuits can allocate voice or data applications in increments of 250 Kbps, and video at 500 Kbps. The maximum amount of QoS for a 768 Kbps circuit is 500 Kbps. T1 circuits can allocate voice and data applications in increments of 250k, and video QoS in increments of 500 Kbps. The maximum amount of QoS for a T1 circuit is 1 Mbps.  At 3 Mbps or above, voice, video and data applications are allocated in increments of 500 Kbps. QoS Maximum Bandwidth Allocations Circuit Bandwidth 512 Kbps 768 Kbps 1.5 Mbps 3 Mbps 6 Mbps 9 Mbps 10 Mbps 12 Mbps Max QoS 250 Kbps 500 Kbps 1 Mbps 2 Mbps 4 Mbps 6 Mbps 6 Mbps 8 Mbps There are similar limits for higher speeds, but the rule of thumb is no more than 2/3 of the connection bandwidth is to be used for QoS. 8.4 CPE & Access Method Requirements 8.4.1 Access Frame Relay and Metro Ethernet Access will include standard dual PE connectivity.  All connections will always be configured with full CIR or CB (committed bandwidth).  Fractional DS3 frame relay connections will be configured at the actual clock rate of the ordered port speed. For instance, a frac-DS3 order for 15mb will be clocked at 15mb – meaning the transmit rate will have a hard limit of 15mb/sec. Private Line access does not automatically include dual PE connectivity as does Frame Relay, ATM and Metro Ethernet. Customers would need to order a second private line per the pricing schedule under "Dual Core Connectivity" and possibly an upgraded router model to accomplish dual PE connectivity. 8.4.2 CPE Currently, FIRN CPE provides two types of approved IOS: Base IP IOS: Does not provide BGP & security features Advanced Security IOS: Provides BGP and security features. (Recommended) For further information on CPE IOS, please visit MFN web site. 9 Additional Services supported through FIRN 9.1 FIRN Secure Internet Bundled Services (Secure Internet Services) 9.1.1 Service Description: 9.1.1.1 A. Secure Internet Services: Secure Internet Services for end users identified in Section 2, are Services combined with a cloud-based basic firewall protection, using a uniform approach and tools, against unauthorized use and access. 1) Contractor shall be responsible for the full lifecycle management of all firewall devices facilitating the Secure Internet Services. This includes: a) Ensuring optimal configuration, tuning, and provide 24x7x365 management and monitoring services using trained and certified security experts; b) Monitoring 24x7x365 firewall subscriptions that protect from network-borne threats using trained and certified security experts; c) Firewall provisioning and deployment; d) Firewall upgrade and patch management; e) Firewall backup and recovery; f) Firewall policy and signature management; g) Firewall policy-based control over applications, users, and content; and h) Auditable and accurate change management logs. 2) The cloud-based basic firewall provides the following security functions for all virtual contexts: a) The Sandbox Analyzer to identify and analyze targeted and unknown files for malicious behaviors. It shall generate and automatically deliver protection for newly discovered malware via signature updates. Signature update delivery shall include integrated logging/reporting. b) Geo Blocking to prevent network based access to internal resources by blocking based on geographic location. c) Application Blocking to identify and block unwanted applications without regard to the port they are using for communication. d) Security Information and Event Management (SIEM): Secure Internet Services will include detailed information provided by the MyFloridaNet QRadar tool. DMS and each Secure Internet Services end user will receive two QRadar login accounts allowing them accurate, correlating information regarding network flows (500:1 sampling), session data, packet captures, reputation white/black listing and endpoint system vulnerability results providing the maximum amount of detail to traffic traversing their network connection. This access shall give Secure Internet Services end users visibility into their Internet connection activity, virtual activity, user activity and application activity, giving them intelligence into their FIRN Secure Internet connection. 3) The cloud-based firewall will provide the following optional more advanced security functions for all virtual contexts subscribing to the Advanced Security and Content Filtering service at the pricing listed in the second column of Table 1.0. a) NextGeneration IPS & IDS: By proactively applying deep packet and application inspection of network activity at the border of the FIRN and the internally protected zones, service will provide better analysis and overall security for each FIRN Organization. Automated correlation and Intrusion Analysis by this service will provide notifications of suspected unauthorized network activity and has the ability to prevent the activity from ever reaching the end user’s internal network. This feature is part of the advanced cloud-based firewall deployment. b) Malware & Anti-Virus detection: This service feature provides real time antivirus and anti-malware protection. End users will have the ability to automatically take action on malicious files currently in transport across the network. This feature will block unwanted malware and viruses at our edge devices before they consume Internet bandwidth or threaten the local network and ultimately desktop endpoint systems users depend on to access the Internet. This feature is part of the advanced cloud-based firewall deployment. c) Next Generation Content Filtering/URL Blocking is enabled upon request. This service helps End users enforce their protection policies and block inappropriate, illegal, and dangerous web content. It will have the ability to block multiple categories of objectionable web content, providing the necessary combination of control and flexibility to protect important resources. The service will deliver sophisticated reporting and visually descriptive monitoring through dashboards, graphs, charts, and data search functionality. This feature is part of the advanced cloud-based firewall deployment. 4) FIRN Help Desk a) FIRN Secure Internet includes access to our standard FIRN helpdesk to provide assistance directly to FIRN end users to answer questions related to all FIRN Secure Internet service tools and services. b) The helpdesk will work directly with the end user to provide advice on remediation methods and industry best practices as they relate to services the contractor provides as part of our Secure Internet offering. c) The helpdesk will be staffed live and/or offer immediate call back within thirty (30) minutes 24x7x365. d) The staff will perform daily “eyes on glass” real-time monitoring and analysis of security events. Monitoring and analysis shall span multiple sources including but not limited to events from the security tools (SIEM), MFN network tools, NetFlow logs, firewall logs, and router logs. e) The MFN/FIRN security team via the FIRN helpdesk will bring any particularly concerning security problems they detect via the SIEM to the end user’s attention for remediation. The helpdesk will also provide remote remediation assistance and advice to the FIRN end users. f) FIRN Help Desk staff via the MyFloridaNet security team will have access to the Contractor’s commercial Security Operation Center (SOC) and threat intelligence research teams to assist in identifying threats and developing preventative countermeasures based on information collected from monitoring events worldwide. 5) End users shall be able to be placed within a specific educational routing domain; also known technically as a virtual routing and forwarding (VRF) instance. In such an enterprise, the State of Florida shall have the capability to secure all schools behind a unified enterprise intranet. 9.1.1.2 B. New Secure Internet and Advanced Security & Content Filtering: Secure Internet services shall be offered based on the rates provided in Table 1.0 below. All current contractual terms, conditions and features of current FIRN Internet Services shall apply with the exception of the following changes: 1) Pricing for this service is flat rate (included in Table 1.0) in the AT&T, CenturyLink and Verizon LATA areas. This new pricing shall be an addition to the flat rate and mileage band pricing originally introduced in the contract or from a previous amendment. The new flat rates in Table 1 may not apply outside of these areas. Any connections outside of the AT&T, Century Link, and Verizon LATAs shall be priced as an individual case basis (ICB). ICB pricing shall never be more than the original flat rate pricing submitted in the original contract. 2) Any FIRN specific core aggregation circuit exceeding 80% peak utilization for more than 2 consecutive 5 minute polling intervals on more than 2 consecutive days will be upgraded within 90 days. Any FIRN end-user provisioned over any MyFloridaNet core aggregation circuit shall follow the aggregation circuit thresholds described in the MyFloridaNet contract. 3) Secure Internet services shall utilize the MyFloridaNet QoS standards and applicable templates. 4) A Contractor managed CPE router is included in the standard service. However, the FIRN end-user may choose to manage the Contractor provided CPE router or provide and manage their own CPE 5) 6) 7) 8) router as long as it is certified by the Contractor. The option to manage the CPE router is at no additional cost to the end user, nor does it alter terms and conditions set forth in the contract or this Amendment. If the FIRN backbone bandwidth is not available in certain areas at time of the end user’s order, and if the connection speed is greater than 2Gbps, the Contractor may provide services via other commercial Internet services at the price in Table 1.0 upon mutual approval by AT&T and DMS. To stay consistent with the intent of this amendment and Erate filing requirements, Secure Internet services will be offered at the nominal bandwidth stated in amendment. At a minimum, ordered bandwidth will be kept consistent for the duration of the FIRN contract or twelve month, whichever occurs first. DMS will only be allowed to increase bandwidth during this period. Bandwidth speeds greater than 1 GB connectivity options are provisioned N x 1GB connections or equivalent bandwidth over 10GB access to a single end user site when and where available. Secure Internet Services Pricing Table Ethernet Bandwidth 10 Mbps 50 Mbps 100 Mbps 200 Mbps 9.1.2 FIRN Secure Internet Bundle Monthly Pricing - Table 1.0 Bundled Cost - Core Additional Cost for + Access + CPE + Advanced Security Basic Firewall and Security Content Filtering and URL blocking $1,121.07 $255.56 $2,053.83 $255.56 $2,605.37 $255.56 $3,622.63 $511.11 300 Mbps 400 Mbps 500 Mbps 600 Mbps 700 Mbps 800 Mbps $4,632.43 $5,162.24 $5,636.83 $6,106.70 $6,372.29 $6,562.81 $766.67 $1,022.22 $1,277.78 $1,533.33 $1,788.89 $2,044.44 900 Mbps 1000 Mbps 2000 Mbps 5000 Mbps 10000 Mbps $6,739.15 $6,914.97 $10,877.40 $15,597.94 $22,688,85 $2,300.00 $2,555.55 $3,833.33 $5,749.99 $8,624.98 Options After hours install, as defined in section 2.4.3 of the MFN Ops Guide, is available at an additional flat rate of $164.25 per device. 9.1.3 Service Requirements A CSAB Order is required to order and establish the FIRN Secure Internet Service. DMS and Customers will not be required to subscribe to FIRN Secure Internet Service. Users shall only subscribe to FIRN Secure Internet Service if they are a FIRN transport subscriber with a FIRN connection at that location. 9.1.4 FIRN NOC Process Refer to section 3 for normal FIRN service process details.  FIRN Helpdesk will escalate advanced security troubleshooting to FIRN SOC according to the standard troubleshooting process 9.1.5 FIRN SOC Process 9.1.6 SLA The FIRN Secure Internet service bundle introduces performance measures via Service Level Agreements for Install, Moves, Adds, Changes and Outages with the following Table 2.0 . Service Performance Measures Table 2 SLA Install, Moves, Adds, Changes (“IMAC”) Performance Target 64kbps to T1 = 60 days >T1 to 45Mbps = 80 business days >45Mbps = 180 business days Liquidated Damages 10% MRC of Service* if performance is not met. Measurement Measured and calculated per incident based on the operational tools provided. Contractor will not be liable where facilities do not exist for access types (excluding Ethernet) greater than 12 Mbps. Site Outage & Service Troubles – Restore Within twenty-four (24) hours Monday – Friday. 5% MRC of the entire service if outage > 24 hours Measured using the trouble ticketing system. SLA clock will start when the trouble has been reported in the ticketing system. The SLA clock will stop when the site has been restored and verified with the end user. For all service troubles, Contractor must open trouble tickets pro-actively and immediately when the outage has been discovered. The time between the actual outage and the opened trouble ticket was opened will be counted towards SLA restoral time. For example: if an outage occurred at 1:00PM and the trouble ticket was opened at 1:30PM, 30 minutes of this time will be counted towards the SLA restoral time. *MRC of Service = MRC of (Core Port + CPE + Access) for each site 9.1.7 MFN NMS Tools Normal NMS tool support will be provided for the base FIRN service and hardware. 9.1.8 Security Information and Event Management Tool 9.1.9 Ordering  Refer to section 4 for details 9.1.10 Billing  Refer to section 5 for detail 9.1.11 FIRN Secure Internet Bundled Services Implementation Process     Orders for the Secure Internet Bundled Services will be submitted through the normal FIRN ordering process detailed in section 4. MFN Security will be notified of order by the Care Center MFN Security will evaluate the customer order according to CSAB options MFN Security will contact customer within 30 days of their order being placed o           All customers who preordered this service (ordered before 7/1/2014) will be contacted and processed in the order in which they were received MFN Security will require each customer to complete and return the Initial Customer Engagement Questionnaire. MFN Security will evaluate each completed Questionnaire and contact customer for additional information as needed. MFN Security will follow-up with specific configuration questionnaires pertaining to the requested features o GEO Blocking Questionnaire o Application Blocking Questionnaire o Security Information and Event Management (SIEM) Request Form o Next Generation IPS / IDS Questionnaire o Malware & Anti-Virus Detection Questionnaire o URL Filtering/Blocking Questionnaire MFN Security will engage additional resources to review overall architecture and complete the final design specifications MFN Security will submit final design specifications to the customer for approval Once the final design has been approved by customer, MFN Security will work with additional teams to establish implementation date Customer will be notified of implementation date for approval MFN Security will initially provision and deploy Security Internet Bundle on agreed implementation date MFN Security will be available to modify implemented configuration for 24 hours after initial turn-up After the initial 24 hour turn-up window has expired, 24x7x365 management and monitoring will be accomplished by the FIRN Helpdesk utilizing established FIRN trouble and change management processes 9.2 FIRN Advanced Security Offerings (ASO) A. ASO can be purchased by end users as an Advanced Security Bundle (ASB) (see B.). Some of these ASB as well as other Advanced Security Offerings may also be purchased separately (See I-J.).. B. Advanced Security Bundle (ASB): ASB includes, for each end user selected location (district headquarters): 1) Fully Managed Device for On-site Intrusion Prevention System (IPS) Device and Service. 2) Fully Managed Device for On-site Premise Firewall Event Logging Management, Analysis and Notification of end user District Area Network (DAN) Firewall. 3) Fully Managed Device for On-site end user Device Event Logging Management and Analysis for up to 15 devices per end user location. 4) Fully Managed Counter Threat Appliance (CTA) to assimilate logging information from all end user selected sources passing on significant events for further analysis. 5) Fully Managed Cloud Based Security Information and Event Management (SIEM) Correlation via forwarded information from the CTA. 6) End User Portal for detailed information regarding their Security incidents and security posture. C. Intrusion Prevention System (IPS): IPS helps eliminate malicious inbound and outbound traffic 24x7x365, without device or signature management, and without increasing in-house headcount. IPS service lets the end user comply with data loss regulations to protect against threats to sensitive data by centralizing the analysis of all devices including firewall logs and provides comprehensive reporting via contractor’s end user portal to demonstrate the effectiveness of the end user’s security controls. The IPS device can be attached to the End User network to provide Intrusion Detection with the onus then on the end user to implement appropriate corrective action. Alternatively, the IPS can be placed in-line of Internet traffic, in which case the contractor shall implement recommended security response to the intrusion. IPS includes: 1) Configuration and implementation. 2) Administration and tuning. 3) 24x7x365 Real-time security event and device health monitoring. 4) Upgrade, change, and patch management. 5) Thousands of unique countermeasures. 6) Daily audits of existing rules. 7) Advanced analysis and blocking techniques, including advanced statistical analysis, suspicious activity correlation and expert security analysis of patterns. 8) Twice weekly countermeasure updates. 9) Intelligence-enhanced threat protection. 10) On-demand security and compliance reporting. D. Firewall Event Logging: Monitoring of any supported end user premise firewall listed below and support for next generation and HA Firewall pairs at no additional charge. Log information shall be incorporated into the provided SIEM and any SIEM indications of a problem are analyzed by security professionals in near real time and end user are notified of any significant firewall events complete with recommended firewall configuration changes. End users desiring a full proactively managed firewall solution can combine this offering with existing FIRN contract firewall management options. Supported firewall devices are: 1) Cisco 2) Juniper Networks 3) Palo Alto Networks 4) Dell SonicWALL 5) Check Point 6) Fortinet E. End User Device Event Logging: The 15 devices can be any mixture of any supported devices (servers, routers, etc.) capable of sending log information to the provided logging device. The logging information shall be fed into the SIEM similar to the Firewall log information and proactively responded to the same way, resulting in notification of the end user of any suspicious activity complete with recommended actions. F. Counter Threat Appliance (CTA): The CTA resides on the end user’s network and shall be responsible for maintaining connections to all sources a end user needs monitored and managed. The CTA shall collect logs from these sources and handles parsing, normalization, de-duplication and filtering of collected events. Security events of interest are sent from the CTA to Contractor’s Security Operations Centers (SOC) via a secured connection, where they are prioritized and, if needed, reviewed by the Contractor’s certified Security Analysts to determine if any malicious or suspicious activity is occurring. Additionally, the CTA is a secure point from which Contractor’s Security Analysts can provide device management. Through the secured connection, the CTA shall have the capability to enable communications and administrative activities for vendor managed devices. G. End User Portal and Reports: The End User Portal shall provide the intelligence and analytics needed to easily understand the risks, demonstrate compliance and make better security decisions. The Portal shall give end users full visibility into their security and compliance posture with advanced reporting functionality integrated across all proffered Advanced Security Offerings. The End User Portal shall include a mobile application ensuring security data is always at the end user’s fingertips. H. Advanced Security Bundled Pricing ASB Monthly Pricing Table 2.0 Bandwidth Monthly Recurring 10 Mbps 50 Mbps 100 Mbps 200 Mbps 300 Mbps 400 Mbps 500 Mbps 600 Mbps 700 Mbps 800 Mbps 900 Mbps 1,000 Mbps 2,000Mbps* 5,000Mbps* 10,000Mbps* $3,413.00 $3,413.00 $3,413.00 $3,717.00 $4,139.00 $4,139.00 $4,139.00 $4,404.00 $4,404.00 $4,404.00 $4,404.00 $4,404.00 $8,808.00 $22,020.00 $44,040.00 *Where available I. Standalone Advanced Security Options. End user may purchase any of the products and services described below. 1) IPS Monitoring is as described in C.3) above. Pricing for those wishing to buy as a standalone product is as follows:ng ‐ Table 3.0 IPS Monitoring Monthly Pricing ‐ Table 3.0 Internet Bandwidth 0 Mbps to 100 Mbps 101 Mbps to 1000 Mbps 1001 Mbps to 2000 Mbps Monthly Recurring $875.00 $1,375.00 $1,550.00 2) IPS Management was included and described in the bundled offering. Pricing for those wishing to buy as a standalone product is as follows: IPS Management Monthly Pricing ‐ Table 4.0 Internet Bandwidth Monthly Recurring 0 Mbps to 100 Mbps 101 Mbps to 500 Mbps 501 Mbps to 1000 Mbps 1001 Mbps to 2000 Mbps 2001 Mbps to 4000 Mbps 4001 Mbps to 10000 Mbps $1,562.00 $2,083.00 $3,250.00 $4,483.00 $6,042.00 $9,042.00 3) End User Device Event Monitoring was included and described in for up to 15 devices in the bundled offering. For those wishing to buy monitoring for additional devices or as a standalone offering pricing is as follows: Device Monitoring Monthly Pricing ‐ Table 5.0 Device Count Monthly 1 15 200 500 Recurring $125.00 $1,500.00 $11,417.00 $21,917.00 4) Vulnerability Management service identifies exposures and weak spots in end user environments by performing highly accurate external scanning and internal scanning across the network. Vulnerability Management shall enable vulnerability scanning without the hardware, software and maintenance requirements of scanning products. Vulnerability results shall be integrated into Contractor’s other Managed Security Services, allowing threats against vulnerable and non-vulnerable systems to be assessed and prioritized accordingly. The Vulnerability Management technology shall be fully managed and maintained by the Contractor’s dedicated vulnerability management team, eliminating administration and maintenance burdens so end users can better focus on protecting assets and reducing risks. Vulnerability Management includes: a) Highly accurate internal and external vulnerability scanning. b) Support for physical, cloud and virtual infrastructure. c) Dedicated vulnerability management team to provide expert guidance and support. d) Flexible reporting and remediation workflow tools via on-demand portal. e) 24x7x365 expert support by certified security analysts. Vulnerability Management service Monthly Pricing ‐ Table 6.0 Network or Server Device Count 128 512 Monthly Recurring $600.00 $1,250.00 1024 Application Count $11,917.00 Monthly Recurring 10 25 160 $767.00 $1,250.00 $3,250.00 5) Log Retention Services shall be a fully-managed service that provides support for a wide range of sources, allowing capture and aggregation of the millions of logs generated every day by critical information assets such as servers, routers, firewalls, databases, applications and other systems. The Log Retention Services shall support hundreds of devices per appliance. Log Retention Services Include: a) Log Retention device with 13TB of compressed storage (3.8TB uncompressed). b) Capturing and storing end user-specified system logs from the IT devices, systems and other network assets to the Log Retention Appliance. c) Implementing software upgrades and security patches to Log Retention Appliance Monitor the information security, system health and performance of Log Retention Appliances 24x7x365. d) Provide end user client access to the Logs. e) Configure any Log Retention Appliance native alerting functionality to provide alerting to notify end user of any such end user Devices no longer transmitting Logs to the Log Retention Appliances. f) Act as the initial point of contact for end user support. Log Retention Services Monthly Pricing ‐ Table 7.0 (13TB Compressed Capacity) End User Device Count Monthly Recurring 25 100 500 Additional 13/3 8TB Capacity $2,225.00 $2,513.00 $3,083.00 $1,250.00 J. Security Incident Response and Consulting: 1) The Incident Response and Digital Forensics practice shall help provide rapid containment and eradication of threats, minimizing the duration and impact of a security breach. Leveraging elite cyber threat intelligence and global visibility, Contractor shall help end users prepare for, respond to and recover from even the most complex and largescale security incidents. The rate is based upon a response tailored to the particular event and is on a per-end user basis. Incident Response Service Monthly Pricing ‐ Table 8.0 Minimum 50 hours Hourly Rated 1 $420.00* *Includes travel and expenses, discounts may be available for additional hours needed during same on-site visit 2) The Contractor’s Security and Risk Consulting (SRC) group shall help customers solve security and compliance challenges. The Contractor shall provide services listed below: Regulatory and Compliance                GLBA (Gramm-Leach-Bliley Act) Gap Analysis HIPAA (Health Insurance Portability and Accountability Act) Gap Analysis FISMA (Federal Information Security Management Act)/NIST (National Institute of Standards and Technology) Gap Analysis PCI (Payment Card Industry) Gap Analysis QSA (Qualified Security Assessor) On-Demand ISO (International Organization for Standardization) 2700x Gap Analysis General Controls Audit Information Security Assessment Security Architecture Review Governance Review Facility Clearance Readiness Review E-Discovery (Electronic Discovery) Security and Compliance Attestation Reporting Third-Party Diligence and Vendor Management IT (Information Technology) Risk Assessment Testing and Analysis          Vulnerability Assessments Penetration Testing Web Application Assessments Network Security Assessment Physical Security Assessment Wireless Network Testing Social Engineering War Dialing Data Discovery and Classification Note: The Gramm-Leach-Bliley Act (GLBA) requires financial institutions to develop, implement, and maintain a comprehensive written information security program that protects the privacy and integrity of end user records. The Health Insurance Portability and Accountability Act of 1996 (HIPPA) includes: the HIPAA Privacy Rule, which protects the privacy of individually identifiable health information; the HIPAA Security Rule, which sets national standards for the security of electronic protected health information; the HIPAA Breach Notification Rule, which requires covered entities and business associates to provide notification following a breach of unsecured protected health information; and the confidentiality provisions of the Patient Safety Rule, which protect identifiable information being used to analyze patient safety events and improve patient safety. Payment Card Industry (PCI) Gap Analysis are designed to combat identity theft and to better secure credit card data. Credit card associations created the Payment Card Industry (PCI) Data Security Standard (DSS) and expect organizations that process, store or transmit cardholder data to comply with these standards. ISO (International Organization for Standardization) 2700x is a series of specifications which include Information Security Management Systems whose focus is based on evaluating process rather than content. These standards contain a Code of Practice consisting of a comprehensive set of information security control objectives and a menu of best practice security controls. Security Risk Consulting Service Security Risk Consulting Service Monthly Pricing - Table 8.0 Minimum 50 hours Hourly Rated 1 $360.20* * Includes travel and expenses 3) All CSAB orders shall include a statement-of-work to be reviewed and approved by DMS and end user. The statement-of-work template shall be defined in the operational and user guide. K. Service Level Objectives: Security Risk Consulting Service Service Level Objectives - Table 9.0 SLO Type Description Action Security Monitoring (applicable to ASB and Standalone options) End user shall receive a response (according to the escalation procedures defined in the End User Portal or in the manner preselected in writing by End user, either through the help desk ticketing system, email, or by telephone) to security incidents within fifteen (15) minutes of the determination by the Service Provider that given malicious activity constitutes a security incident. This is measured by the difference between the time stamp on the incident ticket created by the SOC personnel or technology and the time stamp of the correspondence documenting the initial escalation. A “security incident” is defined as an incident ticket that comprises an event (log) or group of events (logs) that is deemed high severity by the SOC. The most up-todate version can always be found in the Real-Time Events section of the End User Portal). Automatically created incident 1/30th of monthly fee for Service for the affected device Active Health Monitoring (for all FIRN provided devices) tickets (via correlation technology) and event(s) or log(s) deemed low severity will not be escalated, but will be available for reporting through the End user portal. Active health checks identifying the following conditions are subject to the following SLAs: 1/30th of monthly fee for Service for the affected device Device Unreachable – 30 minute response (via phone, ticket, or email) from identification of the device being unreachable. This is measured by the difference between the time stamp on the device unreachable ticket created by the SOC personnel or technology and the time stamp of the correspondence documenting the initial escalation. 5. Annual Affidavit: The Contractor agrees to submit to DMS at least annually, an affidavit from an authorized representative attesting that the Contractor is in compliance with the preferred pricing provision in Section 4(b) of form PUR 1000. 9.2.1 FIRN NOC Process  9.2.2 Refer to section 3 for process details. SLA  Tables will be developed from data in section K of amendment . 9.2.3 Tools  9.2.4 Ordering  9.2.5 Refer to section 4 for details Billing  9.2.6 TBD by Vendor Refer to section 5 for detail FIRN Advanced Security Offerings (ASO) Implementation Process  Being provided by the vendor 10 Appendices 10.1 FIRN Core Layout FIRN Network Diagram 10.2 FIRN Types of Access Layout of the Different Types of Access.doc 10.3 FIRN vs. FIRN DAN Comparison 10.4 NMS Tools Access – CSAB and Access Form Samples EmbarqNotfication.p df New FIRN NMS Tools Access Request Form 10.5 Order Design Reference Documents MFN pricing worksheet.xls Gotcha List.doc MFN QoS Template 10.6 Customer Service Questionnaire MFN Customer Service Questionnaire.doc 10.7 Secure Internet Services Questionnaires FIRN Secure Internet Services-Pre-Questionaire.docx 11 Glossary of Terms This glossary may cover additional terms not found in the Operations Guide, but which may be helpful within the network environment. Word or Acronym Access Circuit Definition The circuit between the Agency and the AT&T “cloud”. ATM Asynchronous Transfer Mode. ATM is a cell switching technology used for Layer2 protocol in ADSL service. CIR Committed Information Rate: The level of data traffic (in bits) which the carrier agrees to handle over a period of time, averaged over a period. CPE Customer Premise Equipment: CPE is equipment at the customer’s location such as the router, CSU/DSU, etc. It can be provided by AT&T or the customer. CSU/DSU Channel Service Unit/Data Service Unit; the device that sits in front of the router and performs line coding, line conditioning, equalizing functions and other similar activities. It can be external (the size of a modem) or internal (WAN interface card). DSL Digital Subscriber Line. DSL is a pair of modems on either end of a single twisted pair wire that delivers ISDN Basic Rate Access. This is commonly used for remote access for Teleworkers, but can also be used as the main access method for an Agency, rather than frame. Frame Relay Frame Relay is a pool of bandwidth made instantly available to any concurrent data sessions sharing the access circuit. This data transport method sends data by dividing it into chunks of up to 8,000 bytes. The chunks are sent one at a time in “rapid bursts”, known as a “frame”. An address frame is sent into the network, which interprets the address and sends the information to its destination at up to 45Mbps. FIRN MPLS Florida Information Resource Network Multi-Protocol Label Switching. The technology that AT&T uses within its network which minimizes lost packets and speeds data transfer using IP protocol. NAT Network Address Translation. An Internet standard that enables a local area network (LAN) to use one set of IP addresses for internal traffic and a second set for external traffic. This allows a company to keep internal IP addresses private. NOC Network Operations Center. The Network managing center which is available 24x7x365 for network maintenance, management and trouble-shooting. . A set of formal rules that describe how to properly transmit data across a network Protocol QoS SLA SME VPN (Quality of Service) - System that provides adherence of SLAs of the Network Service Level Agreement Subject Matter Expert (Virtual Private Network) A method of telecommunication that uses the Internet Backbone as a means of transportation, yet keeps data secure through encryption methods. IOS Internal Operating System used in Cisco routers Word or Acronym Ops Guide Definition Operations Guide, also known as a NOC Guide, is the reference document for the overall management FIRN Network Operations. TFTP Trivial File Transfer Protocol. Common protocol for transferring data. UPS WAN IMG Uninterruptible Power Supply Wide Area Network Incident Management Group 12 Revision History Revision date Version 04/20/2009 1.0 06/03/2014 2.0 09/28/2015 Revised Section Summary of Changes Initial Acceptance 14 Add Amendment 4 Services 3 Updated Escalation Contacts