Security has been getting a bad rap. For far too long the perceived “inhibitors” have been sidestepped by DevOps in an effort to increase productivity.
As Ryan Davidsen, vp, worldwide security solutions, Secureworks, noted, “Traditional approaches for integrating security oversight with application development aren’t keeping pace with the speed required by today’s DevOps teams.”
But as all parties now realize, avoiding security, or waiting until after the fact is time consuming, costly, risky, and not sustainable.
Are there ways to introduce security that won’t slow down the DevOps team? I asked CISOs and other security professionals, and here’s their advice.
1: Shift left. Shift left again. Did I mention “shift left?”
“Shift left as much as possible,” said John Prokap, CISO, HarperCollins Publishers who echoed the most popular advice of security professionals which is to bring automation, testing, education, and code review to the very beginning of the development process.
You’ll move faster and keep everyone happier if you build processes in earlier.
“Changing the rules of development during development is going to slow down any team and cause animosity towards the security team,” warned Jerry Gamblin (@jgamblin), principal security engineer, Kenna Security.
“Developers don’t want to be in conflict with security. They want to address software exposure from the start. That means making security available in the very solutions they use from the onset,” said Maty Siman, founder and CTO, Checkmarx. Read more about Checkmarx’s approach to application security and managing software exposure at the speed of DevOps.
2: Where’s the security? It’s already in there.
CISO for Cabinet for Health and Family Services (CHFS) for Commonwealth of Kentucky, Dennis Leber, bakes in security by making sure checklists, templates, policies, procedures, framework, and requirements are incorporated into a project’s Gantt chart.
“We’re headed towards a world where security enforcement is built into an application at development time and stays with it all the way through production deployment just as it has been done in the application performance management space,” added Steve Herrod (@herrod), managing director, General Catalyst.
3: Everyone bowls better with guardrails.
“By providing common guidance and libraries containing secured functions, security becomes intrinsic and many design flaws and bugs can be prevented,” said Wright.
“DevOps folks need guidance in advance so they know what the right thing is,” added Michael Strong, CISO, GCI General Communication. “No one likes finding out after the fact what other stakeholders’ requirements actually were.”
4: Shields up, captain.
“Build a solid front-end tier level of code that is not always subject to the DevOps model,” said Elliot Lewis (@ElliotDLewis), president and chief architect, Lewis Security Consulting. “Design this layer for security, keep it stable, and DevOps on code behind it. The code on the ‘front line’ is the stuff being attacked by hackers.”
5: Walk a mile in each other’s shoes.
The two groups should be seen as peers.
“When the security infrastructure is automated to the same capacity as the operational infrastructure, then there is no slowdown in the DevSecOps process,” said Rick Howard (@racebannon99), CSO, Palo Alto Networks. “Security is just one more automated module checked in the pipeline of deploying, monitoring, and maintaining.”
6: Bet I can learn this faster than you can.
Levi gamified the security training at his company with an “I dare you to break my stuff” competition. Developers were split up in teams to find vulnerabilities in each other’s code. They were graded on severity, creativity, and volume. Winners received challenge coins and t-shirts that they displayed as badges of honor.
“When you create a secure coding culture, there’s less work at end of the day for dev and sec teams,” agreed CHFS of Kentucky’s Leber.
7: Get yourself a ringer that knows security.
By embedding a security person within the DevOps environment, it fast tracks the previous two tips of educating and cross-pollinating knowledge between the two groups.
“Regardless of what type of environment you have, it is a simple thing to do to invite your CISO to the table at the inception of every project,” said Richard Greenberg (@RAGreenberg), CISO, LA County Department of Public Health. “Need to speed up development? Do it with security on the team. If you bring security in later, it will definitely slow down the entire process.”
8: Test rigorously what’s needed.
Focus testing efforts on where the highest vulnerabilities are. It’s no different than if you were tuning a race car, explained Rushing. Since the engine is the most important part, focus the majority of your testing there. You wouldn’t test everything in the car to the same degree you’d test the engine.
“Same is true for your apps,” said Rushing. “You’d test elements such as shopping carts and payment systems with far more due diligence than you would the rest of the platform.”
9: Hunt down and squash coding flaws… automatically.
“Embed security-aware self-service automation (where compliance, scanning, and testing is a part of the optimized flow) into the continuous delivery mindset,” said Tyson Martin, CISO, Orvis. “It allows cybersecurity and risk to be an advisor vs. a bottleneck.”
“Will this catch everything?” asked Lewis Security’s Lewis. “Depends on whether the scripting is looking for right anomalies.”
To aggressively uncover those anomalies let “security engineers code in security ‘misuse cases’ into the QA testing,” advised Maxime Rousseau (@maxrousseau), CISO, Personal Capital. “For example, QA testing should check if you try 10 passwords, does the account lock-out as needed?”
10: Peer review – the stuff the scanners won’t catch.
While code scans are necessary, they don’t catch all vulnerabilities.
“Make sure developers aren’t hard coding passwords or putting passwords in comments. These are things that code scanners won’t find,” warned Denver Health’s Frietzsche, who recommends reviews from peers, supervisors, or even a third party.
For an organization that’s aggressive about education and automation, this review process will not be permanent. It will only be a stop gap measure.
“Manual gating processes just before release would disappear as the practices for security and particularly security testing are simplified, codified, and automated within the DevOps team processes,” said former QBE Insurance CISO, Mandy.
11: Invite friends to attack your app.
Security Mentor’s Lohrmann is bullish on app soft launches as a means to put a fresh set of eyes and fingers on your app.
He recommends just opening up your app to a wider group of staff members or partners and let them have at it. It’s less formalized than a bug bounty program, which you may want to do subsequently, assuming your company is not afraid of them, said Lohrmann.
12: Don’t store user credentials.
“Apps that store user credentials and other personal identifiable information in the app databases should not be deployed and used,” said Vinay Kumar (@9starinc), CIO/CISO, 9STAR. “Apps should support federated identity and single sign-on authentication. Adopting federated identity and access management and single sign-on prevents apps from storing and learning user’s sensitive personal information including passwords, which in turn, reduces incentive for hackers to target apps/websites/portals for user credentials.”
13: Don’t just scan continuously, authenticate continuously.
Not only should code be scanned, but “programmers should include a credentials check for each time their service is invoked,” said Scott Foote, founder/vCISO, Phenomenati, who also recommends “hardening APIs and service calls, which are a primary point of attack.”
Motorola’s Rushing agreed, “Always test every API for security issues, focusing on least privileged access and only sending data back that was requested, and understanding authentication that is used.”
14: Keep all your secrets in one safe place, not your exposed code.
“With a secure secrets manager, like Chef Vault, you can remove passwords and private keys out of scripts, source code, and other text files,” said Boyle. “If you don’t do something like this, then any API keys, database passwords, private encryption keys, and other sensitive stuff that you put into your DevOps pipeline could be exposed to an attacker or malicious insider.”
AWS offers a secrets manager as well.
15: Get invested in the data.
“Following a data-centric approach to security really helps a DevOps team get the right security controls in the right place,” said SoFi’s Maloney. “Taking the time to understand what data types are being handled by a system, and then to map out where that data is stored, processed, and transferred, can be very useful in identifying the controls needed for data protection.”
“I found it was more innovative to create our own data,” said Bhajaria. “[It allowed us to be] purposeful about what we retained/copied rather than relying on security to protect us from the ‘problem of plenty’ that data often saddles us with,” said Bhajaria.
16: Be where DevOps is – GitHub.
“Security needs to be where DevOps is. In many cases, that’s GitHub,” said Personal Capital’s Rousseau. “My favorite approach is to have static code analysis baked into the GitHub pull request review process. This allows you to be right in the action at the right time with the right insight. That’s the type of agility and automation you need otherwise you’re just in the way. Meet the teams where they are rather than expect them to come to you. Own up and solve for your own security friction.”
17: Think like a bad guy. Create a simple threat model.
“DevOps teams typically focus on building systems that support legitimate and intended actions, meeting functional requirements and achieving desired attributes such as performance and reliability,” said SoFi’s Maloney. “But the potential for fraudulent and unintended actions need to be considered as well.”
“When dev teams have an idea of what threats are applicable to their app and what they need to focus on to reduce risk, that can get addressed during development rather than trying to fix after code is written,” said Ray Espinoza, information security, Amazon.
Shostack recommends playing his Elevation of Privilege game to help you understand what can go wrong. Once you understand potential failures, then dev teams can build features or deploy controls to address problems.
18: Failure feedback direct to developers.
“The best way to stop the cycle of code / release / scan / ticket / repeat is to get the scanning and feedback to the developers faster,” said Robb Reck (@robbreck), CISO, Ping Identity, who suggests cutting security review out of the picture.
“Feedback loops from production to DevOps are critical for understanding if, when, and where application errors are occurring or if portions of the web application are being actively targeted by attackers,” said Ryan Barnett (@ryancbarnett), principal security researcher, Akamai.
“Ideally, security flaws in code should pop up with a red underline like a spellchecker,” said Reck. “Instant feedback is how we change behavior.”
19: Measure, track, and adapt.
“Operations, security, and development all need to understand what is being measured, care about what is being measured, and be passionate about finding how to measure better while resolving the finding together,” said Orvis’ Martin.
Martin’s team measures:
- How many high severity vulnerabilities there are in different lifecycles
- How long those vulnerabilities are open
- The frequency of security scanning and automated testing and what it covers
In addition, Orvis’ team determines a risk level for each application by quantifying the number of attacks per application.
20: Incidents are inevitable. How quickly can you respond?
“The true metric is how quickly applications (and everything beneath them) can be recovered once they have failed,” said Chris Baynham-Hughes (@OnlyChrisBH), head of UK DevOps and RedHat emerging tech, Atos. “An incident is going to happen, so a prevention-only strategy is not enough.”
When you see errant behavior, said Baynham-Hughes, initiate an automated response such as notifying the team, killing a container, and creating a new one. To learn from the attack, collect data for forensic analysis to determine the severity of the issue and how you will handle it in the future.
CONCLUSION: Just add water… and a ton of work to make application security move at the speed of DevOps.
As most of the aforementioned advice recommends, embedding security into DevOps takes time and education for all parties involved. As DevOps was a journey, same is true for DevSecOps.
If all of this advice seems daunting, maybe you should just lean on InfoSec professional Marcus Ranum’s advice to just “hire a brilliant team that understands security and never ever makes mistakes.”
The reality is “building products is hard, and all this DevOps posturing doesn’t change the equation,” said Ranum.
Learn more about Checkmarx’s approach and solutions for managing software exposure at the speed of DevOps.
This article was originally published on CSO Online.