Quelle: CIO USA
ON APRIL 25, PILOT NETWORK SERVICES went out of business,abandoning 200 customers that relied on them for somethingrather important: security. There had long been signs ofPilot´s distress. Customers had recently reported spottyservice from the managed security company. Pilot´s stock,once at $50, had plummeted to 21 cents per share, and Nasdaqdelisted it. Yet this was not some high-flying dotcom thatappeared one day, took some easy venture capital, thenvanished. Pilot was an established, 8-year-old vendor with400 employees and, by most accounts, superior securitytechnology and practices. Its customers included PeopleSoft,VisionTek, The Washington Post Co. and several largehealth-care institutions and banks.
Despite all that, the end came quickly. Pilot employeesreceived four e-mails in rapid succession. The first said thephones would be disconnected. The second added that pagersand mobiles would be taken away. The third said the CFO hadresigned. And for anyone who couldn´t see the elephant---notjust in the room but squirting river water in theirfaces---the last e-mail said, "At 4:30 p.m., you´re fired."
Pilot did keep a skeleton crew to manage customers´ securitythrough the data lines. Responding to desperate pleas fromPilot customers, AT&T suspended the order and kept Pilot´soperations center connected, even though it wasn´t gettingpaid.
With no one watching their networks and an outage threateningat any moment, Pilot customers felt naked. They were suddenlywide open to hackers and viruses. Because some companiesrouted office-to-office traffic through Pilot, they were atrisk of losing secure virtual private network (VPN)connections and remote access. Pilot had hosted entire Webnetworks for other companies, making them even morevulnerable to a complete meltdown.
One such company, Providian Financial, was so distressed thatit sent several IT staffers to man Pilot´s operationscenter. That probably frightened Pilot´s other bankingcustomers, none of whom were expecting a competitivefinancial institution to have access to their networksecurity.
While it´s perhaps the most dire example of failure in theslowing economy, Pilot´s breakdown is not anaberration. Other managed security companies are hurtingtoo. The Salinas Group had already folded. Exodus endured anatrocious first quarter in which CEO Ellen Hancock saideverything was on track "except revenue." Recently, Exodusand another managed security company, Counterpane, announcedthat they are joining forces for efficiencies ofscale. MyCIO.com, once independently operated, was foldedback into its parent company, McAfee. Two other managedservice boutiques, Vigilante and Networks Vigilance, havemerged. "Spending has tapered," says Bruce Murphy, CEO ofVigilinx, another managed security company. "A billiondollars in equity just dried up."
In a matter of days, the managed security services optionturned into a frightening one for CIOs. Until now,outsourcing security management to a boutique company likePilot seemed the best way to go for two reasons: One, that´swhere the most cutting-edge security expertise had migrated,and two, doing security in-house was considered too expensiveand difficult for most companies. But in the wake of thePilot disaster, many CIOs are reevaluating two alternatives:outsourcing their security needs to a large, general servicescompany such as IBM Global Services, or taking care of themin-house.
The problem is that none of the three available options isthe clear winner. Each carries significant risk, and formerPilot customers are trying them all. But all of them agree onone point. Outsourcing security is more work than justwriting a check every month. It´s a full-time job thatrequires in-house resources. Treating it as any less---andmany do---is playing Russian roulette with the entireenterprise.
Small Is Better?
ONE WEEK AFTER THE IMPLOSION, Pilot filed Chapter 7 inOakland Bankruptcy Court. Its website, The Pilot.net, made nomention of the company´s troubles. In fact, the site lookedexactly the same as it had before the collapse. It had aneerie feel, like some Western ghost town.
Pilot´s outage couldn´t have come at a worse time for AnnMarie Durso, CIO of VisionTek, a memory and graphics cardcompany in Gurnee, Ill. She had joined the company in October2000 and was in the thick of a strategic ERP project thatwill help the company launch online retail sales. An outagewould mean revenue losses on online sales, and each daywithout a secure, high-speed connection would add severaldays to the ERP project.
VisionTek has subscribed to Pilot for four years. Like amarriage, the partners just got comfortable talkingless. Security was assumed, and just two months before Pilotwent down, Durso had been baited with a renewaldiscount. Pilot offered to renew her contract at a cut rateif she paid for a full year up front. She did.
"We got blindsided," she says. "We thought [that since] thiswas a provider that had been around since ´96 for us, therewas less of an inclination for us to question them. Butoutsourcing isn´t an abdication. You can´t just hand itoff. Ultimately, the business will hold me accountable, so Ihave to manage the third parties. I have to constantly ask,Are they still growing? Can they handle scale? Are theykeeping their skills up?"
As soon as Durso heard about Pilot, she and her networkmanager, Mike Brown, went from office to office briefingVisionTek´s executives, one at a time, on what the collapsemeant to the company.
"It wasn´t pleasant," Durso recalls about the experience ofhaving to break the news to the CEO, the CFO and thecontroller. Interviewed by CIO the day Pilot filed forChapter 7, Durso was still frayed. "But we´re doing the rightthings. We had a full contingency in place in two days," shesays.
The contingency went something like this: First, get theexecutive staff´s permission to move forward on choosingalternative security providers. Second, create a worst-caseplan. For VisionTek, this meant Brown put his pager on andnever took it off.
Worst case, if AT&T cut the network connections to Pilot,Brown would be paged. He´d box up his servers and drive themfrom Gurnee to downtown Chicago, where an alternativeprovider had offered space and dial-up connections untilVisionTek could find a full-time provider.
Next, VisionTek brought in two ex-Pilot engineers as contractconsultants because they knew Durso´s security better thanshe did. In fact, the day after Pilot went down, VisionTekwasn´t sure of its security status because it had, over time,become Pilot´s responsibility to manage.
Together, the Pilot engineers and Durso figured out wherethey stood and got the network to a point where "we were atleast able to limp along," she says. With security patchedtogether, Durso, Brown and the consultants turned theirattention to evaluating other security vendors. Ironically,she wants a partner similar to Pilot in scope andmethodology. Durso liked Pilot´s level of expertise. Sheliked its 24/7 monitoring. Finding another Pilot with stablefinancials is unlikely. But Durso knows larger companiesoften have less expertise.
Highly sought security talent flowed to the boutiquecompanies for two reasons. First, top IT securityexperts---often from the military and government agenciessuch as the CIA---left public service in droves a few yearsago to start their own companies. Subsequently, venturecapitalists heard tales of Pentagon-level security, so therewas plenty of money out there, until recently. Second, therewas fraternal loyalty; security experts gravitate tocompanies run by their peers.
But the startup trend led to a glut. There were too manyboutiques, and they were burning cash fast. That, in turn,led to aggressive selling, such as Pilot´s offering discountsfor a year´s service for customers that paid upfront. Customers took the deals, which in turn prompted thesecurity vendors to scale up too fast. All of this isprecedented; the ASP market did the same thing two years agoand has stalled ever since.
If small security-only companies can´t escape the economicsof their smallness, the larger general purpose IT servicecompanies can´t get out from under the weight of theirhugeness. Brown evaluated several larger companies and cameaway unimpressed.
"My experience is the bigger companies don´t have theexpertise or the service," he says. "We looked at two ofthem, and it was a circus. They couldn´t even get coordinatedinternally. They hadn´t gotten our business, and they werealready infighting as to who would handle our account."
So for Durso, it becomes a balancing act. She´d like to staywith a security-only company because of the expertise andservice. At the same time, she feels as if she has to slideup the scale to find a stable business. "Really we´re lookingfor a company like Pilot in terms of service," Dursosays. "But you find yourself opting to be more conservative.
"No one has all of the story we want," Durso adds. "You´realways ending up with some kind of trade-off."
As Durso now realizes, outsourcing security is not buyingyour way out of work but rather buying your way intoexpertise and then managing it. But expertise is still thething. She´ll sacrifice only as much of it as is necessary inorder to find a company that won´t go out of business andforget to tell her.
Playing It Safe
ABOUT THE SAME TIME DURSO SHOOK hands on her discount, theCIO at a major health-care organization on the West Coastcalled a meeting with a Pilot executive. This CIO, who askednot to be named because he believes it would paint a targeton his network, had been an early sign-on for Pilot.
About 10 months ago, he watched his service lag and Pilot´sstock swoon at the same time. It gave him pause, so he set upa "frank discussion" with a high-level Pilot executive. Atthe meeting, the CIO challenged the executive on servicelevels and asked direct questions about the health of Pilot´sbusiness and its capability to support him. The Pilotexecutive answered each question, and the CIO was reassured.
Even so, he wasn´t taking chances. After meeting with Pilot,he revisited his contingency plan and now feels fortunatethat he was ready to go when he found out that Pilot was nomore. "We worried," he says. "We probably should have worriedmore. Next time, I´d be even more aggressive."
This CIO´s contingency was relatively smooth. He started witha crude but sturdy frame-relay connection provided byVerizon. Once that was working, he set out to upgrade to ahigh-speed connection also provisioned by Verizon. After thatwas in place, he worked on adding secure access to hisnetwork in the form of a VPN. His e-mail contingency followedthe same slope: first, low-bandwidth access to e-mail, thenhigh-bandwidth access, then secure high-bandwidth access,which brought him back close to what Pilot had provided.
Concurrent to building the network up, he reinserted securityservices into his network while he sought a new managedsecurity partner. He started by assigning one person tomonitor the network, a pale substitute compared with what hewas paying Pilot to do. But it was monitoring nonetheless.
For the first awful week, the CIO had to rely on volunteerex-Pilot and Providian employees, who composed the managementskeleton crew. But within three days, he was out from underPilot, albeit with a temporary structure. "We´re stillsorting it out," he says. "We have some services. We won´thave others like filtering for a while. What we have now isOK."
In choosing his next outsourcer, this CIO echoes Durso as heconsiders the trade-offs between the small vendor with talentversus the big vendor with a stable business. But he´sleaning the other way---toward a bigger company with moregeneralized services. He chose Genuity for his networkconnections. Choosing a managed security provider ispredictably taking longer, but he wants a similarly largecompany, possibly Genuity.
"We´re not interested in breaking in new security vendors. Iwant to see Wall Street firms and large banks on theircustomer list. My ideal would be a large, funded company withdiversified resources," he says. His last requirement is thetricky part---an outsourcing partner has to be "one that´salso highly competent." While the expertise still resides inthe boutiques, the CIO anticipates that large general servicecompanies will start bailing out the smaller companies. Thatway, they acquire the smarts, they have steady bottom lines,and they make security a component of larger managed servicespackages. And indeed, AT&T was ready to buy Pilot but walkedaway at the last moment, several ex-Pilot sources andcustomers say. Symantec has already bought a boutiquecompany, Axent.
If this expertise-through-acquisition scenario plays out assuch, CIOs will have the best of both worlds---stablebusiness and expertise. But that presents otherchallenges. For example, Symantec has products tosell. Partnering with Symantec likely means partnering withSymantec´s products too. And service levels may drop as thesmaller boutiques are subsumed by larger companies.
But for the health-care CIO, less service and expertise isfine. Outages are not.
Three weeks after the incident, with his contingency up andrunning, he says, "We dodged a bullet."
Taking It In-House
NEIL HENNESSY, VICE PRESIDENT OF IT engineering forPeopleSoft, learned about Pilot´s collapse in a most unusualway. At the end of a weekly meeting with his Pilot rep, theman announced, "I have to go back to the office and get firedat 4:30." And in the week leading up to Pilot´s bankruptcyfiling, Hennessy says he was fending off a wake of vultures.
"One guy calls from Southern California, and he´s telling mehow he can offer me everything Pilot did for less money,"Hennessy says. "So I ask him, ´How many employees do youhave?´ and he tells me 40. So I said to him, ´Pilot had400. Why would I trust you?´"
Truthfully, Hennessy had started to lose faith in Pilotbeginning a year before its collapse. He was particularlyworried about the company´s scalability. "They just couldn´tstep up. Not that they didn´t try. Their model was verysecure, but we started looking at other options back then,"he says.
Hennessy´s favored alternative was to phase out his managedsecurity contract and take the task back in-house. This,after all, was the year of uber-viruses and broad,destructive hacks. Hennessy decided that security was justtoo critical to outsource.
His transition plan meshed with his contingencyplan. Hennessy already had a backup carrier with a "dark"data line, one that´s not turned on but could be activated inan emergency. And he started building an internal securitystaff of five with five more to come.
So when his Pilot rep told Hennessy he was going to get firedthat afternoon, Hennessy was able to set the plan in motion,and the transition to in-house 24/7 security was done in fivehours. He credits the quick shift to his engineering team,whom he ranks somewhere between "real strong" and "the bestin the world."
The cost of doing it all in-house was and will continue to bemassive, of course. Hennessy won´t deign to put a number onit, but he readily accepts the fact that he´s paying apremium for in-house security. "It´s definitely far moreexpensive doing it in-house," he says. "On the other hand,there´s far less risk. I´m paying to sleep well."
Why is it more expensive? To begin with, recruiting talent ishard. There´s little out there, and there are plenty ofposers. Some experts put the ratio at about one real expertfor every 10 claiming expertise. Certifications are partly toblame. A résumé with a dozen security certifications mightlook impressive, but it´s misleading. Some certifications aresimply for specific products and teach nothing about bestpractices or security policy. A firewall "expert" might knowhow to configure the box but have no knowledge of whatpolicies should be enforced or even where the firewall oughtto be placed in the context of a specific network.
Paying talent sufficiently is even harder than findingit. Stephen Northcutt, founder of the Global IncidentAnalysis Center and security consultant, says securitycontractors demand up to $500 per hour. Salaries are 5percent to 10 percent higher than what standard IT staffearn.
Keeping talent is the hardest task of all. Northcutt saysmany true security experts are hopping jobs six times a year,upping their salaries $5,000 at each post. Len Cibelli, aformer sales executive at Pilot, expects to get a 20 percentraise from his next employer.
Even so, Hennessy is convinced of the rightness of hisdecision. "We know doing it in-house is more expensive, butwe´ve just decided it´s better than outsourcing," hesays. While talent is thin, Hennessy says a few strongcandidates have come his way due to the economy.
The Hartford Financial Services Group, which was not a Pilotcustomer, has taken many of the same steps asHennessy. Hartford Assistant Vice President of IT JackStoddard outsources little security, only ceding tasks suchas auditing and penetration assessments to outsidevendors. He retains 30 full-time security staff members,tries to recruit the best he can find, pays premiums for themand trains his staff continually. He is adamant about thelimitations of the outsourcing model.
"I don´t see us ever outsourcing," Stoddard says. His CIO,David Annis, believes acquiring and grooming securityexpertise in-house is critical, even if it costs more. Hecalls outsourcing "throwing in the towel." But he understandswhy so many companies do it anyway. Security is so complexand demands such constant reassessments, he says, that doingit in-house requires a "fair amount of redundant duediligence."
Postscript
ON MAY 9, EXACTLY TWO WEEKS after Pilot disbanded, it wasliquidated. There was a Hail Mary as several managed securityvendors tried to take over the business, but thatcollapsed. Emergency operations and support were halted. AT&Tfinally cut the circuits, and Brown at VisionTek received apage. VisionTek was still waiting for the local carrier tosupply a data line, so Brown boxed up his equipment and droveit to its temporary home in a downtown Chicago facility.
On the same day, Pilot´s homepage finally changed itscheerful, "Yes, we´re open" message. "Pilot Networks hasfiled for bankruptcy" was all it said. There were some snippyredesigns of the Pilot website, obviously tacked up by bitterex-employees. The title bar of Pilot.net read: "Pilot NetworkServices is now Imaginary Network Services Inc."
Someone left behind a sarcastic note, which only hinted atwhat had gone on. Anyone who happened by and clicked on"What´s new" would see the note: "Here is the latest aboutPilot: We´re done! Pilot is no more. This company is anex-secure ISP. If it weren´t for being nailed to the perch,it would fall over. Alameda, CA, May 9, 2001."
You can almost hear creaky saloon doors rattling in the windand tumbleweed staggering through the dust.