Quelle: CIO, USA
MATT KESNER HAS been in IT long enough to take silence as a compliment. "People don't often come down to IT to say, 'Nice job," says the CTO of Fenwick & West, a national law firm. "The best you get is that they don't come down at all when things are running well."
But then Kesner tackled the firm's spam problem, and suddenly he found himself a hero. After he outsourced the problem to a managed service provider, the law firm's partners (whose time is worth $350 to $600 an hour) were no longer spending more than an hour a day wading through 300 to 500 spam messages to get as many legitimate messages. "We got quite a few pats on the back and attaboys after putting the spam filter in place. Users saw the difference instantly and are dealing with hundreds fewer messages a day. They actually got excited about it."
Unlike the invisible foe of Y2K, the scourge of spam - which plagues receptionists and CEOs alike - is painfully evident to everyone. Now that spam accounts for 40 percent to 60 percent of most organizations' e-mail traffic, you scarcely need to mention that Ferris Research says spam will cost U.S. businesses at least $10 billion this year, or that Nucleus Research estimates that companies forfeit $874 per employee annually in lost productivity alone. Nor do you have to bring up the fact that spam clogs e-mail systems and siphons IT resources away from legitimate business projects. Spam is a royal pain in the server, and we all know it.
As Kesner has discovered, the sheer ubiquitousness of spam affords CIOs a rare opportunity to look good. Although receiving some spam is inevitable (and employees' expectations should be set accordingly), there's plenty you can do to make things better. In fact, there's plenty you should do, since the problem is only going to get worse, and you can't count on antispam legislation to save the day. (Criminalizing spam would simply drive more spammers to send their messages through offshore ISPs.) Solve the spam problem - or even just put a big dent in it - and you too can be a hero. Here's a look at how otherwise mild-mannered CIOs are leaping into the spam fray to help keep e-mail viable for users.
The Spam Balancing Act
What makes it so hard to write antispam laws or antispam software is that there's no such thing as a universal litmus test for spam. "One person's spam is another person's newsletter," says Eric Ogren, a senior analyst at the Yankee Group. "There's no magic widget the CIO can put in front of the e-mail server and spam goes away."
End users have to be involved in deciding what is spam, he explains, because what's unwanted can vary widely not just from one company to the next, but from one person to the next. What looks like spam to the rest of the world could be essential business communication for certain employees. Colorful language might be important to a customer service agent (displeased customers often lose their tempers, after all), anatomical references may be work-related for a doctor in a research hospital, and Viagra messages could very well be germane to someone in the pharmaceutical industry.
Case in point: When John Zarb, CIO of Libbey, a manufacturer of glassware, china and flatware, tested the Guenivere (a virus and subject-line filter) and SpamAssassin (an open-source spam filter), he had to shut them off after 10 days because they were rejecting important legitimate e-mails. The filters bounce mail with a spam score of 7.5, yet they were automatically assigning 7 points to e-mails from an Asian country in which Libbey has business relationships. Another rule assigned what Zarb calls "bad points" for using all capital letters. Since using all caps is common practice in that Asian country, messages from those business partners easily racked up more than 7.5 points and therefore got zapped. "If the message is a transport document, ouch," says Zarb. His group tweaked the default settings so that Asian e-mails wouldn't automatically accrue so many points. Today, the filters block about 70 percent of Libbey's spam, and Zarb says the false positive rate is far lower but not zero. Because some messages are too critical to miss, he decided to exempt a few employees who deal with international issues from the SpamAssassin filter.
As Zarb quickly discovered, once you start filtering mail, you run the risk of blocking legitimate e-mails because they look like spam. Avoiding an unacceptable level of "false positives" requires a delicate balancing act. Although most vendors will claim they capture at least 90 percent of spam, going above 90 percent will probably result in too many false positives, says Matt Cain, a senior vice president at Meta Group. "You could crank it up and catch 98 percent of spam. But you'd get an unhealthy amount of false positives," he says. "And if you go down to 85 percent, you'll have very few false positives, but too much spam will be getting through."
At printing ink manufacturer Flint Ink, Vice President and CIO Don Barnowski has been trying out Symantec's Norton antispam product. After initially filtering on 300 to 400 keywords, false positives were a daily occurrence. "We started to get calls from people not getting e-mail they were expecting," he says. "That was a red flag; you don't want people questioning the integrity of e-mail delivery."
Cutting the keyword list in half cut the false positive rate in half, but it also let more spam through. "I've accepted the fact that we can't prevent all spam from reaching employees," he says. "Finding out five times a day that I can improve my mortgage rate is irritating but not offensive. There's a big difference there. It's more important to reduce the number of false positives than it is to smother all spam. You can't have it both ways, unfortunately."
To combat false positives, make sure you choose a spam solution that gives you a quarantine area for probable spam that users can access to check for legitimate messages. Users can be alerted in the form of an e-mail digest of all blocked spam subject lines or be directed to a Web mailbox. Outsourcers generally maintain quarantine areas on their servers so that companies don't have to tie up their own networks with suspected spam. Giving end users the ability to add addresses to trusted sender lists (often called whitelists) also ensures that legitimate senders won't get blocked.
"We took the approach of putting in very coarse controls at first, then tightening them up, rather than going with the 'big bang' theory and begging forgiveness for weeks," says Gene Fredriksen, vice president of information security at financial services company Raymond James Financial. "It's absolutely a strategy I'd recommend. You have to build trust in your system first." Fredriksen uses Syntegra's managed service to filter spam for the company's 14,000 mailboxes.
It's also smart to test before you buy, particularly if blocking any legitimate e-mails would harm your business. At Fenwick & West, Kesner created shadow e-mail boxes for some of the firm's biggest e-mail users, into which he put duplicates of all of their messages. He then used those shadow boxes to test antispam products. Because some of the language used in the firm's large commercial transactions - buy, sell, price, dollars - tends to show up in spam, he was dismayed to discover false positive rates of 1-to-1,000 and even as high as 1-to-100.
"In our business, every e-mail from clients is really crucial. We can't block a high percentage of legitimate e-mail," Kesner says. "We needed to be below 0.05 percent, which seemed near unattainable with a filter."
After trying out more than 18 antispam products, Kesner decided to go with Postini's antispam service. With Postini, his false positive rate approaches 1-to-10,000, in part because users can put trusted senders on a whitelist, meaning messages from those senders automatically bypass the filters and get delivered.
Kesner's cautious approach of testing on duplicate messages allowed him to get a real-world read on false positive ratings without worrying about losing any legitimate messages.
The Outsourcing Option
Kesner's testing convinced him that the ability to filter out most spam while maintaining an extremely low false positive rate was worth the risk of outsourcing. "I was cautious of an outside service," he says. "But [being an outsourcer] allows them to respond to spam outbreaks faster than their competitors." Sending out a spam update to thousands or millions of remote users is taxing, so spam software makers tend to roll these updates into packages and send them periodically. A service provider can simply add an update to a few servers in a couple minutes and have the update apply to all customers nearly instantly.
Postini also lets Fenwick & West IT employees choose how much of each kind of spam they want to filter out by setting filters for each of four subcategories of spam: explicit content, get rich quick, too good to be true and racially insensitive. Kesner pays a per-user fee, which turned out to be about half of what he'd budgeted for. And because he's now blocking at least 99 percent of incoming spam (5,000 to 7,000 messages a day get trapped on Postini's servers), Kesner has been able to delay the purchase of four new servers (costing $10,000 to $20,000 each) by more than six months.
Indeed, using an outsourcer can be cheaper than managing the spam problem internally. Water Pik, which manufactures personal health-care products, pool products and heating systems, also found that to be the case. "We looked at the cost of doing it internally, and it was staggering," says CIO Wallace Miceli. "We're talking one or two people full-time," he says. Miceli pays FrontBridge $1.50 per month for each of his 1,000 users, which he says is cheaper than buying and maintaining an onsite filter.
Outsourcing, however, won't work for everyone. Large companies, those with multiple locations whose mail doesn't all pass through one or two points, and those that use both private and public networks, may find it tricky to outsource. And the obvious downside of outsourcing is that it requires giving someone else the authority to decide what e-mail enters your organization. "For a spam filter to work very effectively, it has to look to a certain extent at the body of the message," says John Mozena, a cofounder of the Coalition Against Unsolicited Commercial Email (CAUCE). "Something - even if it's just a piece of software - is reading your company's mail. For some companies, that is not acceptable." Law firms and hospitals, for example, might be wary of exposing confidential client or patient e-mail to a third party.
If you choose to outsource, make sure your service provider will give you timely access to quarantined messages. When Rush Enterprises, a truck, construction and farm equipment dealer, tried outsourcing, Rush's e-mail administrator couldn't see what was being filtered and therefore couldn't tell if the company was missing good e-mails. "When you outsource, you generally lose control," says CIO Scott Kressner. If there was a problem, or if a user needed to be able to receive an important message, it took hours or even a day or two to resolve the situation. Kressner ended up purchasing the antispam appliance (a server loaded with the outsourcer's software that sits in front of the real mail server) and now uses it in conjunction with Symantec Gateway. Although the appliance was more than two or three times the annual cost of the service, Kressner says it's been well worth it to regain control.
A Spam Cocktail
A year or two ago, subscribing to a list of known spammers (known as a black-hole list or a blacklist), or relying on a signature approach (comparing the patterns in a new message against the fingerprints of known spam messages), or using reverse DNS lookup to check whether the sending domain was legitimate might have worked. But companies can't rely on just one type of blocking anymore.
"I'd strongly argue that you need a spam cocktail - a variety of approaches that work together to generate a probability as to whether a message is spam or not," says Meta's Cain. The most reliable products and services subject each e-mail to numerous tests that yield a probability score indicating how likely the message is spam. Companies can then set up rules that, for example, delete messages with a spam score of 95 percent or more, quarantine messages in the 85 percent to 95 percent range, and deliver (with a "suspected spam" warning) messages with scores between 75 percent and 85 percent.
The managed service provided by FrontBridge, for example, uses the cocktail approach. To make it into a user's inbox, an e-mail must clear three hurdles. First, its sender can't be on FrontBridge's proprietary blacklist. Then it must pass through a spam fingerprinting layer that identifies specific characteristics unique to spam. (For instance, spam often hides a stash of unspammy words in white HTML text on a white background to try to fool filters into thinking it's real e-mail; legitimate e-mail would not include white-on-white text.) Finally, it's got to survive a heuristics layer, which involves rule-based scoring. Spamlike behaviors, such as odd characters, spacing or HTML links, earn bad points, which are offset by good points awarded for characteristics that suggest legitimacy. FrontBridge updates 250 of its 10,000-plus rules daily.
Although attacking spam on multiple fronts may seem like overkill, Walter Smith can attest that it's necessary. As director of the global IT infrastructure services group at Advanced Micro Devices (AMD), he calculated that spam was costing the computer chip manufacturer more than $1.5 million a year in lost employee productivity. He first took a crack at handling the problem internally. "Our initial approach was to use fairly simple rules to identify spam and tag junk mail," he says. "We quickly found out that simple rules and spam don't go together." Before long, two full-time employees were consumed with tweaking the rules to account for all of the variations in spam, and even then, they couldn't keep up with the spammers. Only about 30 percent of spam was getting tagged, and some legitimate e-mail was wrongly identified as spam.
So when AMD's e-mail firewall vendor announced an antispam product in May, the decision to use it was more or less a no-brainer, says Smith. AMD already used Tumbleweed both to scan all incoming e-mail for viruses and to prevent confidential competitive information from leaving the company. With the Tumbleweed infrastructure already in place, AMD could plug in the vendor's new spam component for an annual per-user cost of about $5, an investment that paid for itself in less than a month. Today, 90 percent to 95 percent of all incoming spam is tagged as such. And no more than a quarter of a single IT employee's time is needed for ongoing maintenance.
"Having a combination of rules, heuristics and blacklists is really key because of the creativity of spammers," says Smith. "Simple, obvious solutions don't work today. We quickly realized that stopping junk mail is not a core competency of our company. And we needed to get out of that business as soon as we could."
In attacking AMD's spam problem, the last thing Smith wanted to do was to take on the role of corporate censor. "We didn't want to be perceived as content filterers," he says. In the interest of providing a nonhostile work environment, however, AMD does delete all spam with a high probability of containing adult content. But all other spammy mail gets sent along to users, marked as suspected spam. Users then decide for themselves whether to have Outlook filter all spam, put it in a spam folder, or keep it in their inboxes for manual scanning and deletion.
Now that spam is under control at AMD, Smith and his department attained the same herolike status Kesner enjoys. "It's a huge value IT has delivered to the company, and it's been huge, positive publicity for IT," he says.
Act Now, Think Long-Term
Like Smith at AMD, many CIOs would prefer to turn to the same vendor for all of their e-mail security services, including spam filtering, virus protection and denial-of-service protection. "You don't want a box for virus, a box for spam, a box for content filtering, a box for something else," agrees Maurene Caplan Grey, a research director at Gartner. "You want as few boxes as possible, and you want them to work nicely together with a central console for monitoring."
But you shouldn't blindly sign up for whatever antispam solution your current antivirus provider happens to have, warns Meta's Cain. He maintains that the spam offerings of many antivirus vendors are antiquated and not updated often enough to keep up with the spam threat. Keeping pace with spammers has become a full-time job; some antispam outsourcers update their rules daily, hourly or even more often if need be. Your best bet is to invest in a spam cocktail approach from a vendor or service provider with a track record of offering frequent updates (which suggests a commitment to staying current in the spam-antispam arms race) and to make sure that it does not conflict with other e-mail security services. (Ideally, all e-mail services should be integrated.)
While more than 90 antispam vendors stand ready to take your money today, the market will consolidate to about a dozen serious contenders by mid-2004, Grey predicts. She anticipates that the dozen antispam products that survive will be about equally effective, catching 95 to 98 percent of spam with an 0.5 percent false positive rate, even though they may use different technologies to filter spam. She advises choosing a vendor that supports multiple detection methods, and suggests looking at the extent to which vendors are using adaptive technologies (such as Bayesian filtering) that learn about spam's characteristics and can take a more proactive approach to blocking it.
Even though the antispam market is still maturing, you can't afford to wait and see how things will shake out. "Spam is too horrible a problem - and it's going to get more malicious. Two years ago, spam was a little annoyance. If you had a blacklist in place, everything was OK. That's not the case today," says Grey. "You need to do something right now, even though none of this is completely baked."