Is DevOps good or bad for security
With 2016 set to be the year DevOps goes mainstream – Gartner predicts 25 percent of Global 2000 businesses will be using DevOps techniques this year and HP Enterprise is even bolder, claiming that “within five years, DevOps will be the norm when it comes to software development.” Does that mean security problems waiting to happen
Craig Miller, who’s helped move Microsoft’s Bing search engine to continuous deployment (the service now updates four or five times a day), doesn’t believe that’s necessarily true. “The classic response is faster means maybe lower quality or something might get through the process and I don’t think that's true at all,” says Miller. “CD, if you do it right, provides all the auditing you need to actually be confident in the software you push out. You have to make sure your software is high quality and I think security is a subset of quality.”
Forrester analyst Kurt Bittner agrees. “There’s a perception that with DevOps, speed is achieved by cutting corners and skipping important steps, that it’s uncontrolled,” says Bittner. “The exact opposite is true; it’s a very controlled, very structured environment. Doing DevOps right gives you higher quality, better visibility and speed, as opposed to achieving speed by cutting corners.”
That ought to be better for security, but only if continuous integration and continuous deployment are matched with continuous security and monitoring.
The key is the centralized, standardized delivery pipeline that’s a necessary, foundational piece for DevOps, says Bittner. “You get visibility into what's being built and you get the opportunity to inject various kinds of activities; which might be code scanning, or it might be peer reviews, various kinds of security related testing, control over the environment and having the correct settings.”
Miller is uncompromising about the importance of automating testing. “I think is the biggest failure for a lot of companies is that they allow failures in test. We have no tolerance at all for failures.” That might mean changing your tools as well as your development practices, he warns. “The Web security toolset we used was not very scalable; it was an app that somebody ran every week. We’re not going to accept a tool I have to have someone run for me. If it can’t be automated, I think there’s a problem with the design.”
[Related: 7 signs you're doing DevOps wrong]
Bing’s tool for signing binaries during deployment was also manual; both were rewritten, which took time and effort but also improved the tools. “It was the way they did things, but we don’t care how they do things,” says Miller firmly. “We’re not going to do this.”
Recovering fast, and adapting and iterating based on what you’ve learned, are as important as speed to DevOps and Miller dislikes the “fail fast” term. “I like learn fast, because I’m not trying to fail; I'm trying to succeed – but when I don't succeed I want to learn, and hopefully next time it comes around, we know how to do this better now and incrementally get better over time.”
“Putting a guardrail up on the highway allows you to go faster, not slower,” says Alan Sharp-Paul, co-founder of DevOps tool vendor Upguard. “With proper checks, you catch problems before they become showstoppers and security risks in production. And when it’s part of the automated workflow, the overhead is essentially nil.”
That’s what the figures in Puppet’s 2015 State of DevOps Report show as well: “High-performing IT organizations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster.”
The Heartbleed bug in OpenSSL was a good demonstration of that, suggests Bittner. “People who had DevOps and better delivery pipelines were able to respond quickly and that got some attention in businesses; they were able to respond almost immediately and everyone else was scrambling. When a threat occurs, being able to respond quickly is the big differentiator.”
Miller views that as one of the benefits of DevOps. “Because CD emphasizes having a code review process, small check-ins and rapid mitigation come with it. If you can deploy four or five times a day, you can mitigate something within hours.”
The same applies to spotting breaches, says Sam Guckenheimer from Microsoft’s developer tools team. “With DevOps, you're worried about things like mean time to detect, mean time to remediate, how quickly can I find indicators of compromise. If something anomalous happens on a configuration, you have telemetry that helps you detect, and you keep improving your telemetry – so you get better detection, you get better at spotting indicators of compromise and you get better at remediation.”
Continuous deployment makes life harder for attackers in two ways, Guckenheimer explains. “If you're one of the bad guys what do you want You want a static network with lots of snowflakes and lots of places to hide that aren't touched. And if someone detects you, you want to be able to spot the defensive action so you can take countermeasures.”
In a traditional system, with manually configured servers that are all different, there might be vulnerabilities on different systems, and attackers can see admins trying to track or block them. “They're exceptional measures going on in the static slow moving network, but with DevOps you have a very fast automated release pipeline, you're constantly redeploying,” says Guckenheimer. “If you are deploying everywhere on your net it doesn't look like a special action taken against the attackers.”
If DevOps ought to improve security, why doesn't that always happen Sometimes the problem is that companies are using DevOps tools but not really understanding DevOps.
Upguard’s Sharp-Paul says “The primary error we hear about is the attempt to automate before you know what you’ve got. Someone got a budget for DevOps and went out and picked up Puppet Enterprise or Chef and wound up hitting roadblock after roadblock because they don’t understand the actual working nature of their infrastructure. They just bought into the sense of urgency to “do DevOps” and jumped into the deep end without looking.” The security of that approach should be fairly obvious by now.
Guckenheimer agrees. “I don’t think DevOps replaces any good security practices like key management, key rotation, changing defaults or what have you. I do think it changes the pace at which things move; I store more as code and I rebuild the infrastructure all the time, because I’m treating the infrastructure as code whether I'm doing PaaS or Chef or Puppet. That way, you can apply the good practices better. If you ignore them you're vulnerable. The security problems are there and they're exacerbated or alleviated; [with DevOps] I think you have an amplifier.”
There are some common security mistakes. If your build automation means putting your cloud credentials in your code repository or your deployment script, they’re vulnerable. Searching GitHub for AWS and Azure credentials reveals that many people are making the same mistake as Ashley Madison, Uber and D-Link.
Ashley Madison’s leaked code included hard-coded AWS tokens, database credentials, certificate private keys and other credentials. Uber had a database containing personal information about drivers compromised in 2014, after storing the key in a publicly available repo and D-Link recently published its private code signing keys in the open source code for a firmware update. Your cloud credentials are likely to end up subsidizing Bitcoin miners, who scan GitHub for keys and use them to run up hundreds or thousands of dollars of bills.
“What people are not necessarily doing in DevOps which they ought to be doing, is doing good secrets management, using Azure Key Vault and equivalents. What I need to do is keep all the secrets external and as I go from environment to environment though the release pipeline, I need to bind the secrets separately for each environment and make sure those secrets are all external, secure, encrypted and rotated continually,” Guckenheimer says.
Realistically, Azure CTO Mark Russinovich suggests a combination of education to teach best practices and static analysis tools to enforce them. “As much as possible, it needs to be built into the CI/CD [process] so developers don't have to opt in to it, it just happens,” says Russinovich. “You can tell them all day long ‘don't check secrets into code’ but somebody is going to do it, so what you need to do is let them know right then; don't catch it down stream, catch it right when they check it in so that they know they can't do that.”
But that enforcement has to be part an overall DevOps culture, Russinovich says. “What we believe is developers are actually mostly well intentioned. They might forget, there might be someone who hasn't taken training, hasn't taken it in a while, it slips their mind and the gates are just to catch that, because it's too much of a risk not to have them.”
The way the Visual Studio team handles credentials might be a good model. “We run Visual Studio cloud services; we have well over a thousand secrets in there, constantly rotated,” Guckenheimer says. “They’re well rotated and developers do not have access to any of the secrets for production, they're on a separate domain. We practice Just Enough Admin. So were you to somehow get access to the account of the director of engineering – through some back door or social engineering, or brute force it or steal his badge – it wouldn't do you a lot of good, in terms of hacking to get to the production service, because the secrets would be rotated. He would not have admin privileges, because those get enabled on a ‘just enough,’ time-limited expiration basis.”
At best, Russinovich claims, “you might get access to a little bit of source code he could see from his repository – and it would probably be pretty obvious to someone that his account was compromised, and we would shut that off.”
Beyond protecting credentials, DevOps gives you an opportunity to monitor not just the code your developers are writing but third-party code you’re relying on; Forrester’s Bittner suggests viewing that as part of your supply chain management and bill of materials. Monitoring the binary artifacts and components you use, with services like Sonatype and Artifactory, is a relatively novel idea but Guckenheimer predicts it will become common practice, in light of issues like Heartbleed, Poodle and Shellshock.
A more fundamental problem is the common disconnect between DevOps and security.
Getting that kind of visibility about problems and vulnerabilities is part of the way DevOps can help you “shift left” on security, by catching flaws earlier in the development cycle – as well as spotting vulnerabilities in configuration before you deploy to a live system. The potential for security problems and improvements applies as much to infrastructure and operations as to developers, points out Forrester analyst Amy DeMartine, noting that 47 percent of web applications servers have vulnerabilities because they’re misconfigured.
Done properly, she believes DevOps should always include security, but ironically, the reason they don’t is part of the problem DevOps sets out to solve.
[Related: Why are we racing to DevOps]
“I think the reason many companies start DevOps and forget the security folks, is that it's yet another cultural divide,” says DeMartine. “Security people speak a totally different language – breaches, incidents, vulnerabilities and exposures – and everyone puts them at the very end of the development lifecycle, if they’re remembered at all.”
Building compliance and security checklists into the DevOps process stops the security review becoming a bottleneck at the end of the CD pipeline, as well as raising overall quality.
Security teams need “a total mind-shift” to make this work, DeMartine warns. “It’s like the fear of operations that ‘you’re going to make all those changes and you are going to break my environment that’s stuck together with chewing gum and duct tape.’ They have to overcome that fear like operations did. It’s going to be uncomfortable for them, but they’ll get benefits at the end.”
Examples from developers and operations teams of what DevOps has already improved will help that mind-shift, but Bittner cautions, “When we ask a lot of developer teams who have effective continuous integration practices how they engage with their security counterparts, they say they're not ready to have that conversation yet.”
This is ultimately part of the culture change that has to happen as part of DevOps. “The way we're all working today isn't working; it’s not the most effective way.We've set up arbitrary boundaries in orgs between operations, development and security that shouldn't really exist,” says Bittner. “It’s about getting everyone to accept that other teams aren’t the enemy; they’re ultimately trying to do the same thing – they all want the customer to be successful and have good customer experiences.”
CIOs should be prepared for how difficult a change this can be, warns Microsoft’s Miller. “The CIO is going to take heat. This change is a tough one; if you're in the old world and you have custom systems and manual testing, there are going to be problems and you’ve got to be ready. It does demand a certain amount of courage to do CD properly; you might have to rethink your leadership. You certainly have to rethink your process, you have to rethink your expectation from delivery. It's not a trivial path but the payoff is ridiculous. If you’re not doing it, you're risking your business, is the way I look at it now.”
Or as Russinovich somewhat more bluntly puts it, “Devops is coming; it has to have security. Just deal with it.”