Broward County rolls out its own fiber ring, shifts to VoIP
In 2009 Broward County customer complaints regarding the slowness of accessing and using Broward County's services triggered the Enterprise Technology Services' (ETS) Communications group to review the County's network architecture and determine ways to advance it.
The ultimate outcome Broward County's ETS Communications Group implemented an underground fiber optic ring and associated equipment owned and maintained by Broward County which vastly improves performance, saves $780,000 per year and made it possible to embrace IP telephony, a move that saved an additional $28,000 per year.
While the initial need was recognized in 2009, the resulting multi-year upgrade plan didn't get going in earnest until 2010, and the finishing touches were just put in place in 2014. Here's the story.
Our timing at the beginning of the exercise was fortuitous because the ETS' Desktop and Server group had just begun a server virtualization migration and, recognizing how the network upgrade could benefit their effort, offered to join forces.
Working together the teams developed a 3-year strategic initiative to upgrade Broward County to a 10 GigE core network infrastructure. Part of the plan called for reducing complexity and duplication of infrastructure, so the County also decided to converge the voice and data networks and, with voice and data traversing the same circuits, network redundancy would have to be increased because a single line outage could cause a location outage for both critical services.
Of course sound project management is critical when it comes to technology upgrades of this magnitude, so the ETS group created a project team to help identify tasks and resources required to successfully accomplish the goal. The team wrote a project charter and broke the work into manageable chunks to make it easier to proceed.
New fiber backbone
The focus of the project was to replace a costly carrier network with the County's own fiber network. The carrier net consisted of voice and data circuits that were expanding in terms of quantity of lines as well as bandwidth used. As a result, costs were increasing by 15% each year. So we had to provide funding for an ever-increasing service and experiencing constant demands for expansion.
The expense was further compounded by occasional build-out costs that were associated with providing network services to our locations that were not currently served by the carrier's fiber facilities. It could have cost the County tens of thousands of unplanned dollars to subsidize those build outs and resulted in long service delays.
It took us about six months to design our new private fiber optic ring and about four years to build it. The secure backbone runs 41 miles through underground, County-owned conduits and consists of 36 fiber strands. We also upgraded our existing 1Gig gear to new 10Gig optical switches and routers with 10gE interfaces.
The network provides voice, video and data links to 21 buildings throughout Broward County and the agencies they house. It also supports hundreds of County programs and application services to employees and residents, such as Broward.org (our web-site), on-line payments for taxes, licenses, water bills, etc., and everything from E-Permits to the County payroll System and Human Services.
The fiber network can also serve as a connection point for wireless radios and antennas that can be deployed throughout the county to offer wireless Internet services to areas that don't have broadband wireline services today. While we have not yet leveraged the capability, the objective was to build-in the basic interconnectivity infrastructure while we could.
When the project kicked off, the carrier communication services/product contract expired within 18 months of project start. If the network was not installed, tested and operational by the time the contract expired, the County would be on a month-to-month rate at an additional 50% price for hundreds of voice and data circuits. In fact, some services were not available on a month-to-month plan and would require at least a one year contract, committing the County to duplicate costs.
To lessen the risk, we partnered with the Traffic Engineering Division and Florida's Department of Transportation (DOT) to run our fiber at the time they were running new fiber of their own, and they agreed to allow us to use some of their excess capacity if we got into a jam, lessening the contract risk.
The fiber optic project was budgeted at $2.8M, which consisted of a capital cost allocation of $0.25M for engineering services, $0.75M for optical network equipment and $1.8M for construction and infrastructure. As a pragmatic heuristic, 10% of the cost of the fiber would be used for operational maintenance. That would, of course, include fiber cuts, but the only cuts we've experienced to date have been paid by the company that damaged the fiber.
When all was said and done, the fiber network was built for less than $2.5M, some $300,000 below budget, and saves the County about $750,000 per year in line costs. What's more, it set us for another upgrade: a shift to IP Telephony.
IP TELEPHONY
When we crafted the 3-year strategic plan we knew that the County's phone systems would have to be either upgraded or replaced. After analysis, the County elected to replace the systems with a solution that could leverage the new data network infrastructure. Several systems were analyzed for functionality, compatibility, technology, initial costs, operating costs, integration with the email system, and existing data communication infrastructure.
The ETS Communications team ultimately implemented an IP-based telephone system that was homed in two "hardened facilities," one a data center and at a secondary backup site for redundancy.
The IP Telephony solution was less than half the cost of a non-IP-based system. And since the IP voice packets would be carried on the data network, no additional voice lines or circuits were needed. The team formed a model framework for design, installation and management and set out to migrate every building on the fiber network to IP Telephony.
Within three months of project startup the first site was converted to IP Telephony, accommodating 150 users. Today we have 7,000 IP phones and offer soft phones and even video conferencing for everything from recruitment interviews and seminars to construction management.
As old phone systems are replaced or new locations/agencies added, minimal equipment (usually just phones) need to be purchased. The new IP-based telephone system is scalable to tens of thousands of phones.
All told Broward County invested $2.3 million in capital on the IP Telephony infrastructure, which included telephones, applications, licenses, voice mail, call centers and servers for 30 locations. All work installation work and ongoing management is performed by the department so no additional costs were incurred, although we pay approximately $100,000 per year for operations, which includes manufacturer support. This compares to operational costs of more $700,000 per year with the legacy phone systems.
Both projects were delivered under budget, ahead of schedule, exceeded cost savings objectives, increased bandwidth, improved network performance, added redundancy, lessened the time required for installs, moves, adds and changes, and provided for future growth and expansion of Broward County services to its residents.
Due to the forward thinking of the ETS Communications and Desktop & Server groups, Broward County not only got its 10GigE network but also VoIP!
Pat Simes is the Assistant CIO of Broward County, Florida.