Exploring the DevSecOps Toolchain


The authors of the SANS Institute’s DEV540 Secure DevOps & Cloud Application Security course created the Secure DevOps Toolchain poster to help security teams create a methodology for integrating security into the DevOps workflow. As you can see, the poster breaks DevOps down into 5 key phases and includes a massive list of open source tools to help DevOps teams level up their security.

In a new presentation titled Continuous Security: Exploring the DevSecOps Toolchain (Phases 1-2), I spend the entire hour walking the attendees through the first 2 phases of the DevOps workflow:

  • Pre-Commit: Security controls that take place as code is written and before code is checked into version control.
  • Commit: Fast, automated security controls that are invoked by continuous integration tools during the build.

I had the opportunity to present this talk at the Australia Information Security Association’s National Cyber Conference 2018 event this month in Melbourne, as well as a local SANS Community event in Sydney. For those that asked, the presentation slides can be found here:

Continuous Security: Exploring the DevSecOps Toolchain (Phases 1-2)

In the next round, we’ll pick up where we left off and explore Phases 3-4. Until then, cheers!

To learn more about DevSecOps and Cloud Security, check out the DEV540: Secure DevOps and Cloud Application Security course!

About the Author

Eric Johnson is a co-founder and principal security engineer at Puma Security focusing on modern static analysis product development and DevSecOps automation. His experience includes application security automation, cloud security reviews, static source code analysis, web and mobile application penetration testing, secure development lifecycle consulting, and secure code review assessments.

Previously, Eric spent 5 years as a principal security consultant at an information security consulting firm helping companies deliver secure products to their customers, and another 10 years as an information security engineer at a large US financial institution performing source code audits.

As a Certified Instructor with the SANS Institute, Eric authors information security courses on DevSecOps, cloud security, secure coding, and defending mobile apps. He serves on the advisory board for the SANS Security Awareness Developer training program, delivers security training around the world, and presents security research at conferences including SANS, BlackHat, OWASP, BSides, JavaOne, UberConf, and ISSA.

Eric completed a bachelor’s degree in computer engineering and a masters degree in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, GSSP-Java, and AWS Developer certifications.

Taking Control of Your Application Security

Application security is hard. Finding the right people to perform application security work and manage the program is even harder. The application security space has twice as many job openings as candidates. Combined that with the fact that for every 200 software engineers there is only 1 security professional, how do we staff a secure software group and integrate software security across an organization?

There are many approaches for solving this problem being used across the various industry verticals. Most include hiring expensive consultants or conducting long job searches for candidates that rarely exist. Coming from a software development background personally and working with software engineers in various training / consulting engagements across the world has led me to this: capable software security engineers are already working on your engineering teams. Management just needs to put a program in place to educate, empower, and build strong application security champions.

Over the years, I’ve had this conversation with several folks from many different organizations. With many having the same questions and looking for guidance, it became clear to me that this topic would make a great talk.

I had the privilege of presenting the “Taking Control of Your Application Security” talk during an evening session at SANS Orlando 2017 last week. Of course, we had a little fun with a case study and discussed some actionable takeaways for building a culture of secure software engineering. But, most importantly I walked the audience through my journey as a software engineer to an application security consultant.

A number of attendees have asked for a copy of the slides, which can be found here: Taking Control of Your Application Security

For more information on application security training, check out the following resources:

In-Depth Online / Classroom Training: SANS Application Security Curriculum

Developer Security Awareness Training: STH.Developer Training
About the Author
Eric Johnson is a Principal Security Consultant at Cypress Data Defense. At Cypress, he leads web and mobile application penetration testing, secure development lifecycle consulting, secure code review assessments, static source code analysis, security research, and security tool development. Eric has presented his security research at conferences around the world including SANS, BlackHat, OWASP AppSecUSA, BSides, JavaOne, UberConf, and ISSA. He has contributed to several open source projects including Puma Scan, AWS Critical Security Control Automation, and the OWASP Secure Headers project. Eric is also a Certified Instructor with the SANS Institute where he authors several application security courses, serves on the advisory board for the SANS Securing the Human Developer awareness training program, and delivers security training around the world. Eric completed a bachelor of science in computer engineering and a master of science in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.

Threat Modeling: A Hybrid Approach

Editor’s Note: Today’s post is from Sriram Krishnan. Sriram is a Security Architect at Pegasystems. In this post, Sriram introduces a hybrid threat modeling white paper addressing the limitations in traditional threat modeling methodologies.

In the face of increasing attacks at the application layer and enterprise applications moving towards the cloud, organizations must look at security not as just a function, but a key business driver. It goes without saying that it is the inherent responsibility of organizations to provide a secure operating environment for their customers and employees to protect their interest. Understanding the threat landscape is an important prerequisite for building secure applications. Unless people developing the software applications are aware of the threats that their applications exposed to in production, it is not possible to build secure applications. A SANS survey (2015 State of Application Security: Closing the Gap) indicates that threat assessment (which can also be referred to as threat modeling) is the second leading application security practice (next to penetration testing) for building secure web applications. This helps prove that threat modeling is a fundamental building block for building secure software.

Threat Modeling should be viewed as a pro-active security practice to identify security weakness in software applications early Software Development Lifecycle phase. This enables software developers to build software applications by understanding associated threats. There are various techniques published for performing threat modeling such as STRIDE, Attack Tree, and Attack Library. Organizations tend to be prejudiced towards a particular technique while performing threat modeling exercise. However these techniques have their own advantage and limitations in real-world implementation.

The STRIDE technique may be good in enumerating the threats, however STRIDE does not aid in developing countermeasures and mitigation plans. Attack Tree provides an overview about the attack surface at some level of abstraction, which results in not capturing data essential for understanding the threat scenario. Finally, Attack Library may provide information about the attack vectors and be suitable as checklist model, but does not contribute to a complete threat model. When implemented as separate techniques, some of the key aspects required for threat modeling may be missed, thus impacting the productivity and comprehensiveness of the exercise.
To reap the complete benefit of threat modeling, I propose utilizing a combination of modeling techniques to perform the various activities or tasks involved in the threat modeling process. This hybrid model eliminates limitations by adopting a structured approach, capturing optimum details, and representing the data in an intelligible way.

The following white paper analyses the various limitations of the STRIDE, Attack Tree, and Attack Library methodologies and presents a hybrid model that would implement the best techniques from each option.

A Hybrid Approach to Threat Modeling

I have also created a github project that contains a Threat Modeling Template for project teams to use as they get started:

Threat Modeling

About the Author
Sriram Krishnan (@sriramk21) is a cyber security professional with 12 years of experience, currently working as a Security Architect at Pegasystems Worldwide. He has worked with a leading internet service provider, big 4 consulting firm and global banks performing penetration testing, security architecture review, and advising on security best practices for global telecom industry, banking and financial services sector, and government entities.

Continuous Integration: Live Static Analysis with Roslyn

Early in 2016, I had a conversation with a colleague about the very, very limited free and open-source .NET security static analysis options. We discussed CAT.NET, which released back in 2009 and hasn’t been updated since. Next came FxCop, which has a few security rules looking for SQL Injection and Cross-Site Scripting included in the Visual Studio Code Analysis Microsoft Security Rules. Plus, the various configuration analyzer tools that run checks against web.config files, such as the Web Configuration Security Analyzer (WCSA) tools written by Troy Hunt and James Jardine.

Combining these scans all together leaves .NET development teams with a few glaring problems:

  • From past experience, I know that CAT.NET and FxCop have very poor coverage of secure coding issues covered by OWASP, SANS / CWE Top 25, etc. See the presentation slides below for stats from CAT.NET and FxCop scans against a vulnerable .NET target app. The results are very concerning.
  • Aside from the Visual Studio Code Analysis rules, none of the tools actually run inside Visual Studio where engineers are actually writing code. This means more add-on tools that need to be run by the security / development leads, resulting in more reports to manage and import into bug tracking systems. For this reason, it is very unlikely these scans are actually run before committing code, which in my opinion is too late in the development process.

There has to be a better way, right?
Roslyn – The .NET Complier Platform
At the end of the conversation, my colleague asked me if I had tried writing any Roslyn rules yet? Wait, what’s Roslyn? After some quick searching, I discovered that Roslyn, aka the .NET Compiler Platform, is a set of code analysis APIs for the C# and Visual Basic compilers. Using Roslyn, Microsoft provides a way to query and report diagnostics on just about any piece of code. These diagnostic warnings are displayed in two different places:

  • Inside a code document, diagnostic warnings show as spell check style squiggles as code is being written! See the following screenshot for an example:
    Puma Scan diagnostic warning in code
  • In the Error List, diagnostic warnings are added to the standard list of compiler errors. Roslyn provides rule authors the ability to create custom warnings (or errors if you want to be THAT disruptive security team that slows down dev teams) in the list. See the following screenshot for an example:
    Puma Scan diagnostics added to the error list.

After reading Alex Turner’s MSDN Magazine article, I realized that the Roslyn team at Microsoft had provided me exactly what I needed to build a powerful, accurate secure coding engine that runs inside of Visual Studio. During development. As code is written. Before code enters the repository. Bingo!

With that, I decided to help address the gap in the lack of free .NET static analysis tools.
The Puma Scanner
Once we figured out the Roslyn basics, my team and I spent a small part of our consulting hours over the past 4 months learning about the various types of analyzers and building security rules. Of course we had some help, as the folks on the Roslyn team at Microsoft and the Roslyn Gitter channel are very helpful with newbie questions.

Last week, I had the pleasure of presenting our progress at the OWASP AppSec USA conference in Washington, DC. While there is a lot of work left to be done, we are happy to say that Puma Scan has roughly 40 different rules looking for insecure configuration, cross-site scripting, SQL injection, unvalidated redirects, password management, and cross-site request forgery. At this point, I am confident it is worthwhile running the scanner in your development environment and MSBuild pipeline to gain more advanced static analysis compared the free options that exist today.

If you were unable to attend AppSec USA 2016, don’t worry. The hard working volunteers at OWASP record all of the sessions, which are posted on the OWASP YouTube channel. You can watch the presentation here: Continuous Integration: Live Static Analysis using Visual Studio & the Roslyn API session.

For now, as I promised those that attended the presentation, the presentation slides are available on Slideshare. Additionally, links to the Puma Scan web site, installation instructions, rules documentation, and github repository are available below. Let me know if you’re interested in contributing to the project!

To learn more about securing your .NET applications, sign up for DEV544: Secure Coding in .NET!
Puma Scan Web Site: https://www.pumascan.com/
AppSec USA Presentation: Continuous Integration: Live Static Analysis using Visual Studio & the Roslyn API
Github Repository: https://github.com/pumasecurity
Visual Studio Extension: https://visualstudiogallery.msdn.microsoft.com/80206c43-348b-4a21-9f84-a4d4f0d85007
NuGet Package: https://www.nuget.org/packages/Puma.Security.Rules/
About the Author
Eric Johnson (Twitter: @emjohn20) is a Senior Security Consultant at Cypress Data Defense, Application Security Curriculum Product Manager at SANS, and a certified SANS instructor. He is an author for DEV544 Secure Coding in .NET and DEV531 Defending Mobile App Security Essentials, as well as an instructor for DEV541 Secure Coding in Java/JEE and DEV534: Secure DevOps. Eric’s experience includes web and mobile application penetration testing, secure code review, risk assessment, static source code analysis, security research, and developing security tools. He completed a bachelor of science in computer engineering and a master of science in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.

Securing the SDLC: Dynamic Testing Java Web Apps

Editors Note: Today’s post is from Gregory Leonard. Gregory is an application security consultant at Optiv Security, Inc and a SANS instructor for DEV541 Secure Coding in Java/JEE.

The creation and integration of a secure development lifecycle (SDLC) can be an intimidating, even overwhelming, task. There are so many aspects that need to be covered, including performing risk analysis and threat modeling during design, following secure coding practices during development, and conducting proper security testing during later stages.

Established Development Practices
Software development has begun evolving to account for the reality of security risks inherent with web application development, while also adapting to the increased pressure of performing code drops in the accelerated world of DevOps and other continuous delivery methodologies. Major strides have been made to integrate static analysis tools into the build process, giving development teams instant feedback about any vulnerabilities that were detected from the latest check-in, and giving teams the option to prevent builds from being deployed if new issues are detected.

Being able to perform static analysis on source code before performing a deployment build gives teams a strong edge in creating secure applications. However, static analysis does not cover the entire realm of possible vulnerabilities. Static analysis scanners are excellent at detecting obvious cases of some forms of attack, such as basic Cross-Site Scripting (XSS) or SQL injection attacks. Unfortunately, they are not capable of detecting more complex vulnerabilities or design flaws. If a design decision is made that results in a security vulnerability, such as the decision to use a sequence generator to create session IDs for authenticated users, it is highly likely that a static analysis scan won’t catch it.

Dynamic Testing Helps Close the Gaps
Dynamic testing, the process of exploring a running application to find vulnerabilities, helps cover the gaps left by the static analysis scanners. Teams with a mature SDLC will integrate the use of dynamic testing near the end of the development process to shore up any remaining vulnerabilities. Testing is generally limited to a single time period near the end of the process due to two reasons:

  • It is generally ineffective to perform dynamic testing on an application that is only partially completed as many of the vulnerabilities reported may be due to unimplemented code. These vulnerabilities are essentially false positives as the fixes may already be planned for an upcoming sprint.
  • Testing resources tend to be limited due to budget or other constraints. Most organizations, especially larger ones with dozens or even hundreds of development projects, can’t afford to have enough dedicated security testers to provide testing through the entire lifecycle of every project.

Putting the Power into Developer’s Hands
One way to help relieve the second issue highlighted above is to put some of the power to perform dynamic testing directly into the hands of the developers. Just as static analysis tools have become an essential part of the automated build process that developers work with on a daily basis, wouldn’t it be great if we could allow developers to perform some perfunctory dynamic scanning on their applications before making a delivery?

The first issue that needs to be addressed is which tools to start with? There is a plethora of different testing tools out there. Multiple options exist for every imaginable type of test, from network scanners to database testers to web application scanners. Trying to select the right tool, learn how to configure and use it, and interpret the results generated by the tools can be a daunting task.

Fortunately, many testing tools include helpful APIs that allow you to harness the power of a testing tool and integrate it into a different program. Often these APIs allow security testers to create a Python scripts which will perform multiple scans with a single script execution.

Taking the First Step
With so many tools to choose from, where should a developer who wants to start performing dynamic testing start? To help answer that question, Eric Johnson and I set out to create a plugin that would allow a developer to utilize some of the basic scanning functionality of OWASP’s Zed Attack Proxy (ZAP) within the more familiar confines of their IDE. Being the most familiar with the Eclipse IDE for Java, I decided to tackle that plugin first.

The plugin is designed to perform two tasks: perform a spider of a URL provided by the developer, followed by an active scan of all of the in-scope items detected during the spider. After the spider and scan are complete, the results are saved into a SecurityTesting project within Eclipse. More information about the SecurityTesting plugin, including installation and execution instructions, can be found at the project’s GitHub repository:


Using this plugin, developers can begin performing some basic security scans against their application while they are still in development. This gives the developer feedback about any potential vulnerabilities that may exist, while reducing the workload for the security testers later on.

Moving Forward
Once you have downloaded the ZAP plugin and started getting comfortable with what the scans produce and how it interacts with your web application, I encourage you to begin working more directly with ZAP as well as other security tools. When used correctly, dynamic testing tools can provide a wealth of information. Just be aware, dynamic testing tools are prone to the same false positives and false negatives that affect static analysis tools. Use the vulnerability reports generated by ZAP and other dynamic testing tools as a guide to determine what may need to be fixed in your application, but be sure to validate that a finding is legitimate before proceeding.

Hopefully you or your development team can find value in this plugin. We are in the process of enhancing it to support more dynamic testing tools as well as introducing support for more IDEs. Keep an eye on the GitHub repository for future progress!

To learn more about secure coding in Java using a Secure Development Lifecycle, sign up for DEV541: Secure Coding in Java/JEE!

About the Author
Gregory Leonard (@appsecgreg) has more than 17 years of experience in software development, with an emphasis on writing large-scale enterprise applications. Greg’s responsibilities over the course of his career have included application architecture and security, performing infrastructure design and implementation, providing security analysis, conducting security code reviews, performing web application penetration testing, and developing security tools. Greg is also co-author for the SANS DEV541 Secure Coding in Java/JEE course as well as a contributing author for the security awareness modules of the SANS Securing the Human Developer training program. He is currently employed as an application security consultant at Optiv Security, Inc.

Static Analysis and Code Reviews in Agile and DevOps

Editors Note: Today’s post is from Jim Bird. Jim is the co-founder and CTO of a major U.S.-based institutional trading service, where he is responsible for managing the company’s technology organization and information security program. In this post, Jim covers how to perform secure code analysis in an Agile development lifecycle.

More and more organizations are adopting Agile development and DevOps in order to speed up delivery of projects and systems. The faster that these teams move and the more changes that they make, the harder it is for InfoSec to keep up, to understand and manage security risks. In extreme cases where teams are following practices like One Piece Continuous Flow with rapid delivery of individual features, or Continuous Deployment, automatically pushing out 10 or 100 or more changes every day, well-established security controls like manual audits and pen testing are no longer useful.

Security has to be done in a completely different way in these environments, by shifting security controls earlier into the lifecycle, and integrating security directly into engineering workflows.

A key part of this is focusing on code security, through static analysis testing (SAST) and code reviews.

Static Analysis in Agile/DevOps

Self-service, automated code checking with static analysis tools can be wired directly into how engineers write code. Static analysis checking can be plugged into each developer’s IDE to catch problems while they are coding. Fast incremental static analysis checking can be included in Continuous Integration to catch problems immediately after code has been checked in, and deeper static analysis checks can be run as a stage in a Continuous Delivery pipeline to ensure that code won’t be deployed with any serious security vulnerabilities.

Different kinds of static analysis tools provide different kinds of value:

  • Style checking tools like PMD help to ensure that code is readable and follows good practice. These tools won’t make code more secure, but they will make it easier to review and easier to understand and safer to change.
  • Correctness tools like FindBugs catch common coding mistakes and logic bugs which could be exploited or could cause run-time failures.
  • Security analysis tools like Find Security Bugs or commercial tools like HP Fortify can identify serious security vulnerabilities like XSS injection and SQL injection.

To be successful with static analysis, you need to:

  • Ensure that scans run quickly, and find a way to feed problems directly back to developers – into their IDEs or into their development backlog or bug tracking system. Tools like ThreadFix make this easy.
  • Select tools which provide clear, unambiguous information to developers on where the problem is, why it is a problem, and how to fix it.
  • Take time to configure the tools and tune out false positives and low-priority noise – so that developers can focus on real vulnerabilities and other bugs which need to be fixed.

Code Reviews
Code reviews are – or should be – a common practice in many Agile development shops, and are fundamental to the engineering culture in leading DevOps organizations like Google, Facebook and Etsy.

Peer reviews generally focus on making sure that the code is understandable and maintainable, that it follows code conventions and makes proper use of libraries and frameworks. Peer reviewers may also suggest ways to improve the efficiency of code, or ways to make it simpler. These reviews provide a way for developers to learn how to write good code and share best practices. And they can find important logic and functional bugs.

Code reviews can also be leveraged to improve security by finding problems early in development.

Through increased transparency, peer reviews discourage malicious insiders from trying to plant a back door or logic bomb in code. Reviewers can be trained to check for defensive coding (data validation, error and exception handling, and logging) and secure coding practices.

While all code changes should be peer reviewed, high-risk code needs special attention – this is code where even small mistakes can have high consequences for security or reliability. In organizations like Google or Twitter, developers post a request for a member of the security team to review code changes when they assess that a code change is risky. At other organizations like Etsy, the security team works with engineers to identify high-risk code (like security libraries and security features, or public network-facing APIs), and are automatically alerted if this code changes so that they can conduct a review.

Because most changes in an Agile or DevOps environment are small and incremental, code reviews can be done quickly – especially if the code has already passed through static analysis to catch the low-hanging fruit.

As DevOps teams adopt code-driven infrastructure management tools like Puppet and Chef, code reviews can be extended to catch mistakes and oversights in Chef recipes and Puppet manifests, helping to ensure that the run-time is reliable and secure.

Scaling through Continuous Security Checking
Adding automated static analysis checking into a team’s CI/CD workflow, and leveraging code reviews to include security checks will go a long way to improving the quality, safety and security of systems with minimal friction and without adding significant costs or delays. Most of this can be introduced in a way that is almost transparent to the team, without changing how they work or the tools that they rely on.

Instead of depending on point-in-time assessments and control gates late in a project, security checking In DevOps and Agile should be done continuously and incrementally – the same way that these teams build and deliver systems. This will ensure that security can keep up with the rapid pace of change, and will help to scale up your security capability, by enlisting the engineering organization to take on more of the responsibility for code security.

The SANS Software Security curriculum is also excited to announce a new course Secure DevOps: A Practical Introduction co-authored by Jim Bird (@jimrbird) and Ben Allen (@mr_secure). Follow us @sansappsec for more information on this course and training locations in your area.

About the Author
Jim Bird, SANS analyst and lead author of the SANS Application Security Survey series, is an active contributor to the Open Web Application Security Project (OWASP) and a popular blogger on agile development, DevOps and software security at his blog, “Building Real Software.” He is the co-founder and CTO of a major U.S.-based institutional trading service, where he is responsible for managing the company’s technology organization and information security program. Jim is an experienced software development professional and IT manager, having worked with high-integrity and high-reliability systems at stock exchanges and banks in more than 30 countries. He holds PMP, PMI-ACP, CSM, SCPM and ITIL certifications.

2015 State of Application Security: Closing the Gap

The 2015 SANS State of Application Security Analyst Paper and webcasts are complete. This year, Jim Bird, the lead author of the SANS Application Security Survey series, Frank Kim, and I all participated in writing the questions, analyzing the results, drafting the paper, and preparing the webcast material.

In the 2015 survey, we split the survey into two different tracks: defenders and builders. The first track focused on the challenges facing the defenders who are responsible for risk management, vulnerability assessment, and monitoring. The second track focused on the challenges facing the builders responsible for application development, peer reviews, and production support.

Overall, we had 435 respondents, 65% representing the defenders and 35% representing the builders. Based on the results, the communication barriers between defenders and builders are shrinking. But, there is still work that needs to be done:

Defenders and builders are focused on where the greatest security risks are today: 79% web applications, 62% mobile applications, and 53% private cloud applications.

Managers are becoming more aware of how important – and how hard – it is to write secure
software. Today, application security experts are reaching out to builders and speaking at their conferences. As a result, builders are more aware of risks inherent in the same applications that defenders are concerned with.

Management needs to walk the talk and provide developers with the time, tools and training to do a proper job of building secure systems.

For more analysis, the webcasts and analyst paper can be found below:

2015 State of Application Security Analyst Paper: Closing the Gap

Webcast Part 1: Defender Issues

Webcast Part 2: Builder Issues

Thank you to all of the sponsors for bringing this content to the SANS community: HP, Qualys, Veracode, Waratek, and WhiteHat Security.

Also, a special thank you goes out to our webcast panel: Will Bechtel (Qualys), Robert Hanson (WhiteHat Security), Bruce Jenkins (HP Fortify), Maria Loughlin (Veracode), and Brian Maccaba (Waratek).

Happy reading!

About the Author
Eric Johnson (Twitter: @emjohn20) is a Senior Security Consultant at Cypress Data Defense, Application Security Curriculum Product Manager at SANS, and a certified SANS instructor. He is the lead author and instructor for DEV544 Secure Coding in .NET, as well as an instructor for DEV541 Secure Coding in Java/JEE. Eric serves on the advisory board for the SANS Securing the Human Developer awareness training program and is a contributing author for the developer security awareness modules. Eric’s previous experience includes web and mobile application penetration testing, secure code review, risk assessment, static source code analysis, security research, and developing security tools. He completed a bachelor of science in computer engineering and a master of science in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.

DevOps is Killing Maintenance. Let’s Celebrate.

DevOps probably isn’t killing developers.

But it is changing how people think about development – from running projects to a focus on building and running services. And more importantly, DevOps is killing maintenance, or sustaining engineering, or whatever managers want to call it. And that’s something that we should all celebrate.

High-bandwidth collaboration and rapid response to change in Agile put a bullet in the head of offshore development done by outsourced CMMI Level 5 certified development factories. DevOps, by extending collaboration between development teams and operations teams and by increasing the velocity of delivery to production (up to hundreds or even thousands of times per day), and by using real feedback from production to drive development priorities and design decisions, has pulled the plug on the sick idea that maintenance should be done by sustaining engineering teams, offshored together with the help desk to somewhere far away in order to get access to cheap talent.

Agile started the job. Devops can finish it
While large companies were busy finding offshore development and testing partners, Agile changed the rules of the game on them.

Off shoring coding and testing work made sense in large-scale waterfall projects with lots of upfront planning and detailed specs that could be handed off from analysts to farms of programmers and testers.

But the success of Agile adoption in so many organizations, including large enterprises, made outsourcing or offshoring development work less practical and less effective. Instead of detailed analysis and documented hand-offs, Agile teams rely on high-bandwidth face-to-face collaboration with each other and especially with the Customer, and rapid iteration and feedback. Everything happens faster. Customers change priorities and requirements. Developers respond and build and deliver features faster.

Time-intensive and people-intensive work like manual testing and reviews are replaced with automated testing and static analysis in Continuous Integration, pair programming, and continuous review and improvement.

In this dynamic world, it doesn’t make sense to try to shovel work offshore. You have to give up too much in return for saving on staff costs. Teleconferencing across time zones and cultures, “virtual team rooms” using webcams, remote pair programming over Skype… these are all poor compromises that lead to misunderstandings, inefficiencies and mistakes. Sure you can do offshore Agile development, but just because you can do something doesn’t mean that it is a good idea.

Devops is going to finish the job
In DevOps, with Continuous Delivery and Continuous Deployment, changes happen even faster. Cycle times and response times get shorter, from months or weeks to days or hours. And feedback cycles are extended from development into production, letting the entire IT organization experiment and learn and improve from real customer use.

Developers collaborate even more, not just with each other and with customers, but with operations too, in order to make sure that the system is setup correctly and running optimally. This can’t be done effectively by development and operations teams working in different time zones. And it doesn’t need to be.

We all know how outsourcing has played out. In the name of efficiency we sliced out non-strategic parts of core IT and farmed them out to other companies, whether offshore or domestic. CIOs loved it because of the budgetary benefits. Meanwhile, it sparked a thousand conversations about what outsourcing meant for IT, the US economy, individual careers, and the relationship between people and businesses.
But it turned out that we took outsourcing too far. It makes sense for some functions, but it can also mean losing control over management, quality, and security, among other things. Now we’re seeing a lot of those big contracts being pulled back, and the word of the day is insourcing.
InformationWeek, DevOps: The New Outsourcing

Blurred Lines
DevOps intentionally blurs the lines between developers and operations, between coding and support. Engineering is engineering. Project work gets broken down into piece work: individual features or fixes or upgrades that can be completed quickly and pushed into production as soon as possible. Development work is prioritized together with operations and support tasks. What matters is whatever is important to the business, whatever is needed for the system to run. If the business needs something fixed now, your best people are fixing it, instead of giving it to some kids or shipping it overseas.

In DevOps, developers are accountable for making sure that their code works in production:

You build it, you run it

Which means making sure that the code gets into production, monitoring to make sure that it is working correctly, diagnosing and fixing any problems if something breaks.

New features, changes, fixes, upgrades, support work, deployment… everything is done by the same people, working together. Which means that maintenance and support gets the same management focus as new development. Which means that nobody is stuck in dead end job sustaining a dead end system. Which means that customers get better results, when they need them.

Except for enterprise legacy systems on life support, maintenance as most of us think of it today should die soon, thanks to DevOps. That alone makes DevOps worth adopting.

About the Author
Jim Bird (Twitter: @jimrbird), SANS Analyst and lead author of the SANS Application Security Survey series, is an active contributor to OWASP, and a popular blogger on Agile development, DevOps and software security at “Building Real Software.” Currently the co-founder and CTO of a major US-based institutional trading service, where he is responsible for managing the company’s technology organization and information security program, Jim is an experienced software development professional and IT manager, having worked with high-integrity and high-reliability systems at stock exchanges and banks in more than 30 countries. He holds PMP, PMI-ACP, CSM, SCPM and ITIL certifications.

Secure Software Development Lifecycle Overview

In a previous post, we received a question asking, “what is a secure software development lifecycle”? This is an excellent question, and one that I receive quite often from organizations during an application security assessment.

Let’s quickly review the Software Development Lifecycle, also known as the SDLC. The goal of an SDLC is to provide a process for project teams to follow when developing software. A series of steps are completed, each one with a different deliverable, eventually leading to the deployment of functioning software to the client.

Several different SDLC models exist, including Waterfall, Spiral, Agile, and many more. While each of these models is very different, they were all designed without software security in mind. Failing to include software security in the development lifecycle has many consequences:

  •  Releasing critical vulnerabilities to production
  • Putting customer data at risk
  • Costly follow-up releases to secure the application
  • Development teams believing security is someone else’s job
  • And the list goes on…


The requirements phase traditionally allows time for meeting with the customer, understanding the expectations for the product, and documenting the software requirements.  In a Secure SDLC, the requirements phase is where we start building security into the application. Start by selecting a security expert to make security-related decisions for the project team. Send the entire project team through application security awareness training. Ensure the software requirements include security requirements for the development team. Identify any privacy or compliance laws that may impact the product.


The design phase traditionally allows time to choose the platform and programming language to meet the product requirements. The software is broken up into modules, system interfaces are documented, and the overall system architecture is created. In a Secure SDLC, more security-specific steps must be complete. Attack surface analysis and threat modeling will be done to identify potential vulnerabilities, the high-risk interfaces, and any countermeasures. Specify how system authentication will be performed and where secure communication is required.


The construction phase is where we actually start building the product. Development teams build components and libraries based on the design and requirements documents. In a Secure SDLC, provide secure coding guidelines to the development team. Ensure that development team uses the security libraries available in the framework for security-specific tasks (e.g. encoding, validation, data access, authentication, authorization, etc.). As each component is completed, scan the source code with a code analyzer that searches for vulnerabilities, also known as a static analysis tool. The scans present immediate feedback to developers, allowing them to correct vulnerabilities as code is written.


The verification phase is where quality assurance and customers test the product to ensure it meets the requirements. The tests plans typically cover unit testing, integration testing, stress testing, and user acceptance testing. In a Secure SDLC, perform testing to identify vulnerabilities in the live running application. Dynamic analysis, also known as penetration testing, submits malicious parameters to the application in an attempt to compromise the system. Again, providing the feedback to the development team so identified issues can be corrected before product delivery.


Once verification is complete, the product is delivered to the customer. The maintenance phase is where product support and monitoring takes place. Future enhancements are sent back to the requirements phase, and the process repeats itself. In a Secure SDLC, continuously monitor the application for security-specific events and provide feedback directly to the development team. This allows them to detect and react quickly to an attack, taking further action if necessary.


While this post describes a very high level Secure SDLC, the intent is to show you how integrating security into each phase helps the project team understand the importance of security. This understanding helps keep team members engaged during security discussions because they recognize the threats facing their applications. As the software is built, security issues are identified early in process, ultimately delivering more secure products to the customer.

We encourage you to continue exploring the Secure SDLC and visit a number of more in-depth resources on this topic.

The STH.Developer awareness program (http://www.securingthehuman.org/developer) contains over 30 minutes of Secure SDLC awareness training content.

Jim Bird, an expert on Secure Software Development, has written a few in-depth posts on the SANS Software Security blog covering Secure Agile and DevOps proceses.



About the Author

Eric Johnson (Twitter: @emjohn20) is a Senior Security Consultant at Cypress Data Defense, Application Security Curriculum Product Manager at SANS, and a certified SANS instructor. He is the lead author and instructor for DEV544 Secure Coding in .NET, as well as an instructor for DEV541 Secure Coding in Java/JEE. Eric serves on the advisory board for the SANS Securing the Human Developer awareness training program and is a contributing author for the developer security awareness modules. Eric previously spent 10 years working on web and mobile application penetration testing, secure code review, risk assessment, static source code analysis, security research, and developing security tools in the financial industry. He completed a bachelor of science in computer engineering and a master of science in information assurance at Iowa State University, and currently holds the CISSP, GWAPT, GSSP-.NET, and GSSP-Java certifications.

Survey on Application Security Programs – Webcast and Paper

For the second year in a row Jim Bird and I have helped SANS put together a “Survey on Application Security Programs and Practices”. We asked some of the same questions as the previous year, just in a different way. Some interesting trends this year, as taken from the executive summary of the soon to be published paper, include the following:

– There was a significant improvement in the number of organizations implementing application security programs and practices. The percentage of organizations that have an active Appsec program increased from 66% last year to 83% this year—and many of the organizations that do not have a program in place yet are at least following some kind of ad hoc security practices.

– Organizations are testing more frequently. In this year’s survey, more than one-third are doing continuous, ongoing security testing of their applications, whereas only 23% indicated doing so in our previous survey.

– Organizations continue to face the same kinds of challenges in getting management buy-in for application security programs. But the leading inhibitor for putting effective Appsec programs in place is now a shortage of application security skills, whereas in last year’s survey, the leading inhibitor was management buy-in and funding. In this year’s survey, organizations also ranked technical resources to maintain security in production their fourth most difficult problem.

To find out more please register for our complimentary webcast on Wednesday, February 12 at http://www.sans.org/info/150770

If you register for the webcast you’ll get an advance copy of the paper that will be published in the SANS Reading Room at http://www.sans.org/reading-room/analysts-program

To find out more about Software Security Awareness training for developers please visit SANS Securing the Human at http://www.securingthehuman.org/developer. Information about longer developer security training courses is available at https://www.sans.org/courses/developer.