Related Articles
Building high-volume, complex information systems is hard. But when time is limited and the client is the federal government, as we learned most recently and most painfully with the failed launch of Healthcare.gov, it’s nearly impossible.
Yet with only six months remaining on the contract with its longtime supplier, the Federal Communications Commission seems on the verge of repeating past mistakes by choosing a new vendor to manage the mission-critical system that lets consumers take their existing phone numbers with them when switching services.
KAREN BLEIER/AFP/Getty Images
The Number Portability Administration Center (NPAC) is a common database that US and Canadian carriers, including VoIP providers, can query when routing calls. More than 2,000 carriers rely on the center. Since its inception in 1998, NPAC has been operated by Neustar, whose current contract expires in June.
By all accounts, the system has worked exceptionally well. According to Neustar, the NPAC databases are queried more than a million times a day and maintains up-time of 99.999 percent. The company has successfully completed 11 major software upgrades.
But the system’s success can perhaps best be measured by the fact that consumers now take for granted the ability to keep their existing phone numbers when changing carriers or devices or when switching from wired to wireless services. As many as 80 percent of US households have cut the cord to landline telephone service altogether in favor of mobile phones — an evolution accelerated by number portability.
Recognizing the likely disruption of switching to a new provider and a new system, the FCC in 2010 began the process of revisiting the contract, promising to sign a new contract by late 2013.
The process has been plagued from the beginning, however, by delays, bidding irregularities and a near total lack of transparency.
Finally, in April, a stakeholder group made up of the largest carriers, trade associations and state regulators known as the North American Numbering Council (NANC) recommended that the FCC ditch Neustar in favor of the only other bidder, a subsidiary of Ericsson called Telcordia, which also goes by the name iconectiv. (Neither Ericsson nor iconectiv responded to requests for comment on this story.)
Why shut off an IT system that’s working?
The FCC has yet to act on the recommendation, but will soon.
After the NANC’s recommendation was accidentally posted on the FCC’s website in June, the agency belatedly invited public comment, giving commenters only two weeks (PDF) to respond. Very few of the key documents were made available, however. All of them were subject to a protective order, meaning prospective commenters would have had to register with the FCC and travel to Washington to view them.
The fundamental objection is obvious: Why switch to a new system that has yet to be built from a new vendor?
Neither the NANC nor the FCC will disclose how the industry participants arrived at their recommendation, but insiders believe the carriers’ main complaint about Neustar is cost. Roughly half of Neustar’s 2013 revenue, or about $450 million, came from fees approved by the FCC and paid by telephone companies to use the system.
If Neustar is charging too much, however, that is a very poor reason to undertake a high-risk transition to a new and still-unbuilt replacement system and vendor.
Neustar has complained of other irregularities in the bidding process that add to the risk. Telcordia was permitted to submit its bid after the original deadline, for example, and the agency was never clear how many rounds of bidding there would be (just one, as it turns out). Eligibility requirements were not established until after bidding closed, and there is still serious doubt whether Ericsson’s existing vendor relationships with North American carriers disqualify Telcordia from an explicit requirement that the NPAC provider be a neutral party.
The industry participants acknowledged, moreover, that they prepared no written analysis to support their recommendation. Smaller carriers and public safety representatives have complained of being largely excluded from the process.
There is also a strong likelihood that key technical requirements for the new system can’t even be defined. For example, the FCC earlier this year OK’d trials by wireline telephone providers to begin the shutdown of the old switched network. That transition will ease the dwindling number of customers onto all-digital networks — the most dramatic revolution in the history of voice communications and one with a significant, yet still undetermined impact on local number portability.
Trials now under way include tests of new numbering systems that will ensure continued portability and test ways to improve the system for an all-digital world. The FCC only held its first workshop on numbering trials in February last year.
Another federal IT meltdown in the making
Taken together, the delays and other failings of the NPAC bidding process raise a strong possibility of another Healthcare.gov-like meltdown.
That, in any case, is the assessment of Stagg Newman, the FCC’s former chief technologist. In an op-ed earlier this year for Roll Call, Newman bemoaned still more deficiencies of the process, including the FCC’s failure to ask prospective bidders to demonstrate a working system or provide a transition plan as part of their bids. Both have long been considered standard requirements for large IT system proposals.
“From the outside looking in,” Newman concluded, “the selection process appears to have been driven by lawyers and policy makers. Where is the objective, technology-driven process needed to ensure success for the millions of consumers who rely on number portability?”
Newman’s concerns are well-founded. Despite the dangers involved in a transition, there is no indication that the FCC will take into consideration the high likelihood of delay or outright failure as it considers the industry group’s recommendation to switch. Though bidding documents and other key communications have been kept secret, the agency’s request for proposal barely mentioned risk management or contingency plans should the vendor fail to complete a replacement system in time.
And the RFP assumed the selected vendor would have years, not months, to design, build, test and implement its system and interfaces.
So perhaps the most worrisome feature of an imminent portability crisis is that minimizing the biggest risks doesn’t seem to have been part of the selection criteria used by the industry participants or the FCC. Nor is there any indication that the agency intends to submit Telcordia’s plan to an independent IT expert to assess those risks and identify strategies for managing them.
A general risk assessment (PDF) for a new NPAC system was performed in January by the Standish Group, an IT analysis firm that studies large-scale systems development projects. Based on comparisons with similarly complex projects, the report found that replacing the current system in one “big bang” conversion, as the FCC requires, had only a 6 percent chance of succeeding on time and within budget. The report concluded:
The Standish Group does not see any real value in replacing Neustar with a new NPAC vendor except for possible cost savings. Cost savings usually is not a good reason to replace a specialized mission-critical service. … It is our opinion that a change in vendors will be more likely to cause increased costs and no savings.
Telcordia acknowledges Standish’s credentials but objects to the lack of specifics in the report, admitting at the same time that the details of its proposed new system have been kept secret. In a blog post last January, Chris Drake, CTO of Telcordia’s number portability business unit, wrote that “the report relies mainly on generalities and examples from completely different areas and offers very little insight into the specific circumstances of the NPAC procurement.”
(A company spokesperson for Neustar reconfirmed that Neustar funded the Standish report.)
Perhaps, as Telcordia argues, having a sole provider for this critical system was never a good idea. But that was a decision the FCC made under pressure many years ago, when a second developer of the initial databases failed to deliver a working system.
And nothing about the current bidding process would correct that error. It would merely shift it to a different sole provider.
The bottom line is simple: If the FCC is seriously considering replacing a complex and fully functioning system with software that has yet to be designed, coded or tested, the agency should first prove wrong the alarming projections of the Standish Group and others.
Which is easy enough to do. Just hire a neutral party with expertise in large-scale systems development and integration, and give them access to Telcordia’s proposal.
Given everything that has gone wrong already with bidding for a new NPAC system, that’s the least the FCC can do.
