Ask the Expert – Jim Manico

Jim Manico is the VP of Security Architecture for WhiteHat Security, a web security firm. Jim is a participant and project manager of the OWASP Developer Cheatsheet series. He is also the producer and host of the OWASP Podcast Series.

1. Although SQL Injection continues to be one of the most commonly exploited security vulnerabilities in the wild, Cross Site Scripting (XSS) is still the most common security problem in web applications. Why is this still the case? What makes XSS so difficult for developers to understand and to protect themselves from?

Mitigation of SQL Injection, from a developer point of view, is very straight forward. Parameterize your queries and bind your variables!

Unfortunately, mitigating XSS can be very difficult. You need to do contextual output encoding in 5 or more contexts as you are dynamically creating HTML documents on the server. You also need validate untrusted HTML that is submitted from widgets like TinyMCE. You also need to parse JSON using safe APIs such as JSON.parse. And then you need to deal with the very challenging issue of DOM Based XSS, a challenge that even tools have a problem discovering. And this problem is getting worse in the era of rich internet application development.

2. What’s the real risk of XSS – what can attackers do if they find an XSS vulnerability? How seriously should developers take XSS?

Attackers can use XSS to set up keyloggers, deface a website, steal session cookies or other sensitive data, redirect the user to an untrusted website, and circumvent CSRF protections … just to get started. If developers want to build secure web applications they NEED to take XSS defense seriously. And it can be quite difficult to accomplish this – especially for modern RIA/AJAX applications.

3. Where can developers and testers and security analysts go to understand XSS? What tools can people use to prevent XSS today and where can they find them?

One of the best XSS prevention guides is the OWASP XSS Prevention Cheat Sheet. Over 500,000 hits and counting.

For advanced practitioners there is the OWASP DOM XSS Prevention Cheatsheet as well.

4. When is XSS going to be solved for good, or will we have to keep on living with the risk of XSS exploits for a long time?

If developers are forced to manually output encode ever variable, I feel XSS will always be with us.

But there is hope in standards.

Content Security Policy 1.1 is a W3C draft  which promised to make XSS defense a great deal easier on developers. There is only mixed browser support for CSP today, but in 2-3 years when all browsers full support the CSP standard, there will be a browser-based highly effective AntiXSS methodology available to all.

I’m also fond of the HTML5 iframe sandboxing mechanism for XSS defense.

Ask the Expert – Dan Cornell

Dan Cornell has over fifteen years of experience architecting and developing web-based software systems. As CTO and Principal, he leads Denim Group’s security research team in investigating the application of secure coding and development techniques to improve web-based software development methodologies. Dan was the founding coordinator and chairman for the Java Users Group of San Antonio (JUGSA) and currently serves as the OWASP San Antonio chapter leader, member of the OWASP Global Membership Committee and co-lead of the OWASP Open Review Project. Dan has spoken at such international conferences as RSA, OWASP AppSec USA, and OWASP EU Research in Greece.

The cost of fixing software bugs has been studied for a long time now, with experts like Capers Jones collecting data from development and maintenance projects around the world. But up until recently there has been very little data available on the cost of remediating security vulnerabilities. Denim Group is one of the few companies doing analysis in this area, collecting and analyzing data from security remediation projects.

1. Can you please explain how much and what kind of data you’ve collected so far, and how long you have been collecting it? What makes this statistical data unique?

We do a lot of software security remediation work where we actually go into applications and fix vulnerabilities. What we started doing was collecting the level-of-effort data for those projects in a structured manner so that we could start to look at how long it was taking our folks to fix different types of vulnerabilities as well as what the composition of vulnerability resolution projects looks like. The data we have released reflects the results of looking at 15 different remediated applications and spans about a year and a half. I don’t know of any other analysis that has been done on this topic. The folks at Veracode and WhiteHat have released some really cool data about the prevalence of different classes of application vulnerabilities as well as the calendar time they persist, but I haven’t seen anything released about the level of effort required to fix those vulnerabilities.

We see organizations go through lots of scanning activities and that leads to giant piles of vulnerabilities that they then have to fix. You can benchmark yourself against the data from WhiteHat and Veracode to see if you have more or fewer vulnerabilities than other organizations and that’s great. But it doesn’t help you answer your next question: “How am I actually going to fix this stuff and what is it going to cost me?” Our data can be really useful for security analysts and development team leads who find themselves in this situation because we provide a framework for these remediation projects and some numbers that can be used to start roughing out project plans.

 2. Based on this data, what are the easiest security problems to fix – the low hanging fruit that people should go after first?

Based on our data, the three easiest types of issues to fix are: removing unused methods, fixing logging that goes to the system output stream and addressing empty catch blocks. But let’s look at this in a little different way. Just because something is easy to fix doesn’t mean that you necessarily want to fix it first or even fix it at all. You need to have an understanding of the risk associated with these vulnerabilities, as well as the budget and other resources available for remediation. Once you understand your constraints, you can start to ask, “How do I best apply these limited resources?” Our data lets teams run different scenarios and put together potential project plans.

 3. On the opposite extreme, what are the hardest problems to fix, the problems that have the most risk or cost the most to fix, and why?

Based on our data, the three issue types that took the longest to fix were SQL injection, reflected cross-site scripting and redundant null checks. But again you have to be careful how you look at this data. One of the things we were really surprised to see was the amount of time it took at address SQL injection issues – by far, the longest of any vulnerability type at 97.5 minutes per vulnerability. When we drilled into the projects a bit deeper we found that we had some particularly nasty SQL injection issues that we addressed in the sample. So that SQL injection number was far higher than it would have been if we had a larger sample of fixed vulnerabilities that more closely reflected the “average” SQL injection vulnerability. Also, just because something is “easy” or “hard” doesn’t mean that you automatically fix or not fix the vulnerability. You need to gauge the risk that vulnerability exposes your organization to and maximize the impact of your remediation investment.

4. Are you seeing any changes in this data over time? Are some problems getting simpler and easier to fix – or harder?

We haven’t actually tracked this data over time, but what we did track the project composition of these remediation projects. People think about fixing vulnerabilities and expect it to be “easy,” but for these projects we were fixing multiple instances of several different classes of vulnerabilities. These were software maintenance projects. And in a software maintenance project, you don’t just spend “keyboard time” making changes – you also have to set up your development environments, and you have to verify that your security fix was successful and that the application still works, you have to deploy the remediated code into production. There’s also the overhead and just general friction that go along with any project in a large enterprise.

We looked at what percentage of these projects were spent in each phase and what we found was that the time spent actually fixing vulnerabilities varied from 59% in the “best” case all the way down to 15% at the lowest. I think that data is potentially more valuable than the vulnerability-specific data because it starts to provide insight into what you need to do to make the process of fixing vulnerabilities less expensive. One project spent 31% of its time just setting up the development environment. This was an end-of-lifed application everyone was scared to touch, but the organization had identified some vulnerabilities that they could not live with. If the organization had maintained a proper run book or other deployment documentation it would have been far cheaper to fix the identified vulnerabilities. Looking at the composition of the different projects starts to make you see how general development best practices like automated testing and scripted deployment reduce the costs of fixing vulnerabilities.

We see teams using the data in two ways – the vulnerability-specific data can be used to construct bottom-up project estimates for remediation projects and the project composition data can be used to sanity-check the final project plans to make sure that sufficient time has been included to accommodate all of the ancillary tasks that are absolutely necessary, but that teams tend to forget about when they’re planning these projects.

My RSA presentation with the results of this research can be found online here:


Ask the Expert – Johannes Ullrich

Johannes Ullrich is the Chief Research Officer for the SANS Institute, where he is responsible for the SANS Internet Storm Center (ISC) and the GIAC Gold program. Prior to working for SANS, Johannes worked as a lead support engineer for a Web development company and as a research physicist. Johannes holds a PhD in Physics from SUNY Albany and is located in Jacksonville, Florida.

1. There have been so many reports of passwords being stolen lately. What is going on? Is the password system that everyone is using broken?

Passwords are broken. A password is supposed to be a secret you share with a site to authenticate yourself. In order for this to work, the secret may only be known to you and that particular site. This is no longer true if you use the same password with more than one site. Also, the password has to be hard to guess but easy to remember. It is virtually impossible to come up with numerous hard to guess but easy to remember passwords. As a result, users will reuse passwords, or use simple passwords that are easy to guess. Most users would have to remember several hundred hard to guess passwords to make the password system work.

2. What are the best practices today for managing and storing passwords for applications?

As recent breaches have shown, passwords may be leaked via various vulnerabilities. As a first step, you need to ensure that your applications are well written and as secure as possible. Code with access to passwords needs to be reviewed particularly well, and only limited database users should have access to the password data to avoid a SQL injection in one part of your code that doesn’t need access to the password data from being escalated to affect password data.

Password data should not be stored in clear text, but encrypted or hashed. Most applications never need to know the clear text form of the password, making cryptographic one way hashes an ideal storage mechanism for passwords. But it is important to recognize that any encryption or hashing scheme is still vulnerable to offline brute force attacks once the data is leaked. After a password database is leaked, you can only delay the attack, but you can not prevent it.

A proper hash algorithm should delay the attacker as much as possible, but should not be too expensive to cause performance problems for the application. Any modern hash algorithm, like sha256 will work, but you may need to apply it multiple times in order to slow down the attacker sufficiently. Other algorithms like PBKDF2 and bcrypt are slower and may be preferred over the rather fast and efficient hash algorithms like sha. But it is important to use an algorithm that is well supported by your development environments. Applying a fast algorithm like sha256 multiple times will in the end be as efficient as using a more expensive algorithm once (or fewer times).

In addition, data should be added to the password before it is hashed. This data is frequently referred to as “salt”. If possible, each password should use a different salt. The salt serves two purposes. First of all, multiple users tend to pick the same password. If these password will be hashed, the hashes will be identical. An attacker breaking this hash will be able to access multiple accounts. If a unique “salt” is added first, the password hashes will be different. In addition, by using a sufficiently large salt, brute forcing the password will be harder. One of the most efficient brute force methods is the use of “rainbow tables”. Rainbow tables are pre-computed hash databases that are compressed to allow for very efficient storage and queries. Currently, these tables are widely available for strings up to 8 characters long. If one restricts the table to more common dictionary terms, larger tables are available as well. By adding a long unique salt, the string to be hashed will exceed the length commonly found in rainbow tables, and by picking a non-dictionary salt, and attacker would have to create specific rainbow tables for each salt value, making them impractical. The salt is stored in clear with the account information. Sometimes, it is just prepended to the hash. A random salt is ideal, but in many cases, a simple salt may be derived from information like the user id or the time stamp the account was created at.

As a basic summary describing secure password storage: Take the users password, prepend a salt unique to this account, and hash the result as many times as you can afford using a commonly available hashing algorithm.

Over time, you should expect CPUs to become more powerful, and as a result, the number of iterations required to protect your password will increase. Design your application in such a way as to make it easy to change the number of iterations.


3. How can people test or review their apps to make sure that they are handling passwords in a secure way? What should they look for?

Quick Answer: When reviewing an application, make sure that authentication is implemented only once and easily changed later to adopt to a more complex hashing scheme. Developers should be instructed to use these authentication libraries and refrain from writing code that accesses the password database directly.

Ask the Expert – John Steven

John Steven is the Internal CTO of Cigital. John’s expertise runs the gamut of software security from threat modeling and architectural risk analysis, through static analysis (with an emphasis on automation), to security testing. As a consultant, John has provided strategic direction to many multi-national corporations, and his keen interest in automation keeps Cigital technology at the cutting edge.

This is the last in a series of interviews with appsec experts about threat modeling.

1. Threat Modeling is supposed to be one of the most effective and fundamental practices in secure software development. But a lot of teams that are trying to do secure development find threat modeling too difficult and too expensive. Why is threat modeling so hard – or do people just think it is hard because they don’t understand it?

“Effective in what regard?” The world’s conception of what threat modeling is, what it produces, and what it accomplishes remains in flux. Many find appeal in thinking through “who may be able to do what to your system where” but we as a community can’t even agree on the vocabulary. I addressed this issue in a blog post. Microsoft, security “big spender”, published a threat modeling book and tool and still suffers divergence between the STRIDE+DREAD camp and its security testers on what it means to threat model. Without clear and consistent direction: what it is, what goes into the process, and how you use its output; threat modeling will remain hard.

Difficulty buttresses threat modeling misunderstanding. The security community understands vulnerability discovery fairly well. We’ve systematized techniques and automated with tools (for a fun distraction, start with Gunnar Peterson’s thoughts on checklists). Unfortunately some assessment gurus continue to couch threat modeling as the vulnerability discovery box labeled “think”: threat modeling becomes the catch-all for hard stuff we’ve not yet been able to bake into a tool or checklist.

And, yes, threat modelers operate at distinct disadvantages. Modelers almost never possess adequate source material (requirements, a design diagram, or entitlement data). They’re often forced to operate within an aggressive release schedule as well.

Summarizing: It can be tough, security experts aren’t doing the best job of providing support, and downward pressure on time/LoE squeezes quality.

2. What are the keys to building a good threat model? What tools should you use, what practices should you follow? How do you know when you have built a good threat model – when are you done?

My clients and I have surveyed a plethora of open source and commercial tools. We agree that they often create more drag than the lift they provide. So, with deference to good work that SD Elements has done, I don’t recommend starting with a tool: it’s distracting.

Combat threat modeling ambiguity with a “Threat Traceability Matrix”. Create your own simple spreadsheet tool if you like. Simply document:

  • WHO – threat: an agent capable of doing you harm
  • WHAT – misuse/abuse a threat intends to promulgate
  • WHERE – attack surface on which a threat will conduct a misuse/abuse
  • HOW – specific attack vectors (what –> how creates an attack tree)
  • IMPACT – negative effect on a business objective or violation of policy
  • MITIGATION – cell to track risk acceptance, transfer, or mitigation

Managing this table throughout the threat modeling, vulnerability discovery, and risk management loop will serve you well remaining largely agnostic of threat modeling religion. I like Cigital’s threat modeling–it’s very software and design centric, with roots in risk. Others like a more asset-centric approach. IMPORTANT: Use whatever practices effectively add high-value “whos” “whats” or “hows” that your vulnerability discovery techniques currently omit.

Combat difficulty and LoE/cost constraint by recognizing threat modeling will be an iterative and incremental process. Your context will govern whether you decide to go depth-first (into a threat, or a component) or breadth-first across the whole system/threat-space.

You’re probably never done. As you add to or refactor your system, your threat model’s fidelity decays. Update a threat model when new attacks are discovered (what/how), or when motivation/skill/access amongst your adversaries (the “who”) change. Your risk management framework (RMF) governs when you stop.

3. Where should you start with threat modeling? Where’s your best pay-back, are there any quick wins?

0) Draw a diagram – Give developers & modelers a ‘one-pager’ to work from in concert. Start with the attack surface. Focus on the software, not just systems & network.

0) Organize your work – Start by drawing up the traceability matrix.

1) Cut-and-paste – When you threat model application Y pull in application X’s threat model and evaluate applicability of its content.

2) Produce output that someone else consumes – Identify attack vectors (“hows”) that can be explored with other vulnerability discovery techniques (dynamic testing, code review, and even design review).

3) Pass-by-reference – Keep “what” enumeration conceptual. If attacks vectors fall into the AppSec canon don’t spend too much time describing “hows”. There’s little pay-back in re-treading what your tools and penetration testers discover well already.

4) Model what’s changing – Projects start green-field only once. The rest of the time you’re changing or extending existing functionality. Rather than stopping everything to model the complete system, start with what’s changing in this release and those key components on which changes depend. This confines scope and keeps work relevant to current development.

Ask the Expert – James Jardine

James Jardine is a senior security consultant at Secure Ideas and the founder of Jardine Software. James has spent over twelve years working in software development with over seven years focusing on application security. His experience includes penetration testing, secure development lifecycle creation, vulnerability management, code review, and training.  He has worked with mobile, web, and Windows development with the Microsoft .NET framework. James is a mentor for the Air Force Association’s Cyber Patriot competition. He currently holds the GSSP-NET, CSSLP, MCAD, and MCSD certifications and is located in Jacksonville, Florida.
This is the second in a series of interviews with appsec experts about threat modeling.
1. Threat Modeling is supposed to be one of the most effective and fundamental practices in secure software development. But a lot of teams that are trying to do secure development find threat modeling too difficult and too expensive. Why is threat modeling so hard – or do people just think it is hard because they don’t understand it?
I believe there is a lack of understanding when it comes to threat modeling.  Developers hear the term thrown around, but it can be difficult reading in a book or an online article how to actually create a threat model.  Without a good understanding of the process, people get frustrated and tend to stay away from it.  In addition to the lack of understanding of threat modeling, I believe that sometimes it is a lack of understanding of the application.  If the person/team working on the threat model doesn’t understand the application and the components it talks to, it can severely impact the threat model.

Threat modeling is time-consuming, especially the first iteration of a threat model, tied into the first time the team has created a threat model.  In the development world that time equates to costs in which there is no direct return on investment.  Developers are under tight timelines to get software out the door and many teams are not given the time to include threat modeling.  The time factor all depends on your experience, size of the application, development process and it will get quicker as you start doing the threat modeling on a regular basis.  The cost of doing the threat modeling, and adhering to the mitigations during development, will be less than the time spent going back and fixing the vulnerabilities later.

2. What are the keys to building a good threat model? What tools should you use, what practices should you follow? How do you know when you have built a good threat model – when are you done?

Motivation is the biggest key to building a good threat model.  If you are forced into threat modeling you are not going to take the time to do a good job.  In addition, an understanding of what you want to accomplish with your threat model.  Some threat models use data flow diagrams, while others do not.  You need to identify what works for your situation.  As long as the threat model is helping identify the potential risk areas, threats, and mitigations then it is doing its job.  It is important to set aside some time for threat modeling to focus on it.  Many times, I see it is something that is a task that is buried with other tasks handed off to one developer.  This should be a collaborative effort with a specific focus.

There are many different tools available to help with threat modeling. Microsoft has the SDL Threat Modeling Tool which is a great tool to use if you don’t have any threat modeling experience.  Microsoft also has the Escalation of Privileges card game which is meant to help developers become familiar with threat modeling in a fun way.  You could just use Microsoft Excel, or some other spreadsheet software.  It is all up to what works for you to get the results that you need.

Threat models are living documents.  They should not really ever be finished because each update to the application could affect the threat model.  You can usually tell if you are done with the current threat model when you are out of threats and mitigations.  Think of making popcorn, when the time between pops gets longer, the popcorn is probably done.  The same thing with threat modeling, when you haven’t identified a threat or mitigation in a while, it is probably finished.

3. Where should you start with threat modeling? Where’s your best pay-back, are there any quick wins?

In theory, threat modeling should be done before you start actually writing code for an application.  However, in most cases, threat modeling is implemented either after the code is started, or after the application is already in production.  It is never too late to do threat modeling for your applications.  If you are still in the design phase, then you are at an advantage for starting a threat model following best practices.  If you  are further along in the development lifecycle, the start high level and then work your way down as you can.  Often times, the threat model will go a few levels deep getting more details.  Each level may add more actors and more time.  You can start with just the top level identifying inputs/outputs and trust boundaries which will be a big security gain.  Then as time progresses, you can add a level or additional detail to the current level.

If your company has a bunch of applications that are somewhat similar, you can create a template for your company threat models that can help fill in some generic stuff. For example, protecting data between the client-server, or server-server.   This should be in all of your apps, so you can have a base template where you don’t need to do all the legwork, but just verify it is correct for the current application.  This can help increase the speed of threat modeling going forward.

Ask the Expert – Rohit Sethi

Rohit Sethi is a specialist in building security controls into the software development life cycle (SDLC). He has helped improve software security at some of the world’s most security-sensitive organizations in financial services, software, ecommerce, healthcare, telecom and other industries. Rohit has built and taught SANS courses on Secure J2EE development. He has spoken and taught at FS-ISAC, RSA, OWASP, Secure Development Conference, Shmoocon, CSI National, Sec Tor, Infosecurity, CFI-CIRT, and many others. Mr. Sethi has written articles for InfoQ, Dr. Dobb’s Journal, TechTarget, Security Focus and the Web Application Security Consortium (WASC), has appeared on Fox News Live, and has been quoted as an expert in application security for ITWorldCanada and Computer World. He also created the OWASP Design Patterns Security Analysis project.

1.  Threat Modeling is supposed to be one of the most effective and fundamental practices in secure software development. But a lot of teams that are trying to do secure development find threat modeling too difficult and too expensive. Why is threat modeling so hard – or do people just think it is hard because they don’t understand it?

I’ve recently started touring across North American OWASP chapters facilitating discussions with the topic “Is there an end to testing ourselves secure?”. It’s an open dialogue about the problems with successfully scaling early-phase secure SDLC activities, and one of the hottest topics is threat modeling. There are a handful of companies that have had success rolling out threat modeling with appropriate tool support. Unfortunately, most people at these OWASP chapters are saying they just do not have the organizational expertise, motivation, and available cycles to really institute proper threat modeling. Many software shops lack a single architecture diagram, let alone the capacity to build data flow diagrams. They’ve found it difficult to enumerate every use case, to identify interaction between processes, and to enumerate defenses against generalized classes of threats such as spoofing.

For threat modeling to scale, it’s supposed to be something that a development team can do on its own without the support of information security. The reality is that unless you have a development force that’s very well educated in security (i.e. generally places where senior executives see application security as a differentiator),  developer interest is generally low and they often feel that they don’t have the requisite expertise to effectively build a threat model.

2. What are the keys to building a good threat model? What tools should you use, what practices should you follow? How do you know when you have built a good threat model – when are you done?

At Security Compass and SD Elements we tend to advocate a more agile approach to threat modeling, ‘Threat Modeling Express’, which is just a name we gave for something that companies have been doing for years. In the absence of a solid investment in a threat modeling library, we think you need application security expertise to build useful threat models. The approach is basically a 2-4 hour collaborative review session between an app sec expert, a developer, and somebody who represents the business. You enumerate a few major uses cases, think about threats from a business process perspective (i.e. “I’m worried that somebody can view somebody else’s shopping cart ”), collectively rate the risk of these threats, excuse the business person and try to hash out the technical attack vectors that somebody might use to enable these threats.

Ideally you spend more time focusing on domain-specific (a.k.a. business logic) threats because the domain-agnostic threats  won’t very much between use cases. It’s an informal approach, it’s not very comprehensive, it’ll almost certainly miss things, but it’s much easier to get started with until you’ve built up a business case to do comprehensive threat modeling. Most importantly, it requires a relatively low time commitment, which is almost always the biggest blocker to doing threat modeling.

3. Where should you start with threat modeling? Where’s your best pay-back, are there any quick wins?

The earlier you can start the better, and doing it right at application design is most cost effective. The reality is that in most cases you won’t have any choice: you’re brought in to do a threat model and development is either well under way or done. In these cases it’s better to treat threat modeling as a precursor to a security assessment. For example, effective threat modeling should identify and prioritize domain-specific threats, which gives a penetration tester or a source code reviewer guidance on where to focus.

In terms of quick wins you’ll find that domain-agnostic threats tend to become repetitive for  classes of applications. For example, almost all web applications have to build controls for XSS and CSRF. These recurring security issues dovetail into a discussion on security requirements, which I recently wrote an article for InfoQ. If you can institute a solid library of reusable threats for particular application types, then you can focus threat modeling time on the all-important domain-specific issues.

Ask the Expert – Nick Galbreath

Nick Galbreath is director of engineering at Etsy, overseeing groups handling fraud, security, authentication and internal tools. Over the last 18 years, Nick has held leadership positions at a number of social and e-­commerce companies, including Right Media, UPromise, Friendster, and Open Market, and has consulted for many more. He is the author of “Cryptography for Internet and Database Applications” (Wiley), and was awarded a number of patents in the area of social networking. Today he shares his insights on Continuous Deployment and security.

1. In Continuous Deployment, developers push software to production several times a day. Please explain how this reduces risk in development and operations.

Before we talk about risk, let’s talk about what Continuous Deployment requires. It needs both a mechanism and a policy for success. The mechanism is being able to deploy quickly without some manual process (i.e. that guy who magically deploys code for you). Every organization needs this now. The policy is how, what and when you choose to deploy. This is highly company and risk dependent, and you’ll need to adjust this to your needs. You are unlikely to want to deploy, say, database schema changes as often as CSS changes due to the risk involved.

However one important part of the policy is emphasizing “small changes.” For complex Web-based, data-driven systems, there is no amount of QA in your development environment that can guarantee that the code won’t cause an operational or security issue. None! So given that, would you rather have a big bang change with dozens of authors and hundreds of files being modified each with multiple change sets? Good luck rolling that back and debugging when things go wrong! Or would you rather have a few very small changes that can be code-reviewed “by inspection” and is easy to undo, each with an audit trail. To me the latter is much less risky for both site availability and security.

Also note that while there is risk in making changes, there is also risk in not making changes. Not being able to push out timely security fixes and patches leaves your organization exposed during the (long) release cycle. With a Continuous Deployment process, exercised multiple times per day, you know you can push a security fix when the need arises.

2. Many Appsec professionals are concerned that Agile development teams build software too fast to be secure. Continuous Deployment seems to accelerate this even more. How do security controls and checks fit into Continuous Deployment, and what controls and checks need to be done differently to keep up with the pace?

I’ve seen no study saying that agile methods are any better or worse for security than any other methodology — security can be ignored equally well in all methodologies! Waterfall with full SDLC has been producing security flaws for decades, so I’m not sure that is the gold standard. I suspect that all methodologies, properly implemented, have about the same change-to-bug ratio in a release (i.e. number of lines of code changed to number of new bugs created in a single code deployment).

Continuous Deployment is about movement of code from development to production. It doesn’t mean you don’t do review for security, architecture and operations and don’t write tests and don’t do peer code reviews. You still need to do all that. Continuous Deployment also means holding developers more responsible for their code quality, because developers can immediately see the results of their work. There is little to no buffer between them and their code running.

As mentioned to make this work, developers need to learn an “incremental programming” style involving pushing lots of small changes. This is not natural to most developers and takes a while to learn. A lot of the changes in a proper Continuous Deployment process do nothing. This dark code isn’t executed in production. It just sits there. Why? Since we know if something does go wrong on deployment we can rule out these dark changes, focusing our efforts on code that actually does something. In the meantime, that dark code is now in source control for all to see and review, and we know has compiled and passed basic automated QA tests. For code that is active, there are a number of strategies to manage risk. Lighting up the dark code is normally done under a configuration flag, so it’s easy to identify the change and to easily turn off if something occurs. Features are “ramped up”, by first testing internally, then exposing to 1% of the site’s users, then 10% and so on to make sure everything is working right before going to 100%. If a problem is found, it can just as easily be ramped down.

So far I talked a lot of risk and development. As for security and Continuous Deployment, the summary is “secure by default” and “alert on when it’s not.” It’s easier said than done. The following aren’t really specific to Continuous Deployment, but perhaps they are more important:

  • Isolate, insulate or ban functions that are easy to misuse. Cryptographic functions are a prime example of this. Most of these APIs are too low-level, so they’ll need wrapping into something that mere mortals can use. Then alert on the raw function. If new crypto starts showing up in the code base, it probably needs review (and ideally this code was already reviewed in your software development life cycle).
  • Segment out sensitive code and alert on changes to it. Your password storage mechanism shouldn’t be changing that often. If it is, find out why.
  • Use static analysis, even for dynamic languages. Prevent those silly bugs before they go out into production. In C, those silly bugs are frequently security vulnerabilities. Static analysis for dynamic languages such as PHP is becoming more popular as well (e.g. see
  • Making testing easy for developers and automatic for deployment. Set up a continuous integration stack such as Jenkins and have it run every time. This also means spending time on making your tests run fast. If they are slow, no-one will want to run them.
  • Make untrusted user inputs secure by default. Right now, every web platform gives you the raw user input. Unless you explicitly escape it, it’s insecure. This is impossible to manage. If possible flip it around so inputs are escaped, and if you need the raw data, you have to explicitly un-escape the data. In other words, the inputs are secure by default. This is complicated and probably deserves its own paper.

3. Where do teams that want to move faster and use Continuous Deployment need to start if they want to do this in a safe and secure way? What tools and practices do they need to have in place to succeed?

To get started with Continuous Deployment, the very first thing to do is to “put a button on it.” Get your release engineer (or whoever deploys code) to start automating the release process so it’s a “one button” process or simple shell script that anyone can do. The release engineer should not view this as job-threatening as it actually makes him more valuable. Then work on making the time it takes to deploy shorter and shorter. If it hasn’t been done already, this process is likely to cause a whole cascade of site operations improvements: automation, standardization, monitoring of your stack. All of which help improve security. Then work on how fast you can upgrade key servers (e.g. Apache HTTPD) or even the operating system. You want all of this to be painless, since if it’s not, you’ll postpone it, or worse not do it all and miss a key security patch.

A basic requirement is to log all changes: what is the change, who made the change, and who pushed out the change to production. You’ll probably want some conventions around when people can push and on peer review. You’ll also want to make visible site operations and when site changes are happening. You’ll want to visibly correlate changes to problems.

The previous section answered what is needed to make Continuous Deployment work securely before code goes out. Now it’s time to see what happens after code is deployed. Again, none of these are unique to Continuous Deployment:

  • Make security visible. Graph potential SQLi and XSS attacks. These are normally quite common, and turns security into a visible event for all to see. It’s a fantastic tool for security education, as well as knowing when someone new is probing you.
  • Use attacker-driven testing. Use previous results to guide how you do manual or semi-automated testing. You’ll find attackers don’t scan everything but focus on particular regions of your website. Maybe they found something.
  • Monitor core dumps. Your server shouldn’t be core dumping very often, if ever. If it is, maybe you need to patch or upgrade. Or maybe someone found a buffer overflow.
  • Monitor “server 500” errors. Can you reproduce them? Probably it’s just a QA problem but maybe it’s someone scanning your system and they found something.
  • Monitor database SQL syntax errors. SQLi breaches don’t happen in a vacuum. An attacker needs to spend a lot of time probing your application for the appropriate entry point. These often end up generating SQL syntax errors.
  • Look for other anomalies. New really long URLs coming in? New parameters showing up in a query string? Maybe a code review needs to happen.

It takes a while to get the technology, the code, and the culture ready for secure Continuous Deployment. Perhaps the most important effect of Continuous Deployment is the transition of operations and security teams typically known for blocking change and “saying no” into service organizations to enable change.

Ask the Expert – Chenxi Wang

This is the second in a series of “Ask the Expert” articles where we chat with leaders in the software development and application security space. Our guest is Chenxi Wang, Ph.D., who is Vice President and Principal Analyst at Forrester Research. A leading expert on content security, application security, and vulnerability management, Chenxi leads the effort at Forrester to build the application security and Web 2.0 security research portfolio.

Chenxi will be delivering the keynote at the SANS AppSec Summit in Las Vegas on May 1. Here are her thoughts on application security.

1) How big is the AppSec problem that we are all facing today?

In my opinion, Application security is the number one problem in the security industry today. Doesn’t matter how good your security processes are, if you have one critical vulnerability in your code, all bets are off. Until we can consistently produce better and more secure code, the security problem will not getter fundamentally better.

So, in short, the AppSec problem is huge and is one of the most under-rated aspects of computer security.

2) The software community is made up of a lot of smart people. Why haven’t we been able to solve the problem of writing secure software?

It’s an economics problem. Until software consumers are willing to pay more for secure code, software producers will not invest significant resources. Think about it, the most brilliant technology minds of our generation are working on how to better serve ads to users – simply because the Internet advertising market is a huge market and remains largely untapped still.

3) Is the problem solvable? Is it really possible for developers to write secure software? If so, where should developers and businesses start? What are the first changes that they need to make?

I believe the problem is solvable, but not without significant changes to mentality, processes, and technologies. I believe the most efficient place to start is for companies to establish the right processes and incentive structures to foster behaviors towards generating secure software artifacts. This includes having the right collaboration structure (a small, elite set of software security experts working with a satellite group of associates who are software engineers themselves), processes and gatekeeper functions to adopt software security measures in development.

I also think current technologies has a lot of room for improvement. Innovations are needed to make automated analysis technologies more precise, more accessible to developers.

Ask the Expert – Jeremiah Grossman

This is the first in a series of “Ask the Expert” articles where we chat with leaders in the software and application security space. Our first guest is Jeremiah Grossman who founded WhiteHat Security in August 2001. A world-renowned expert in Web security, Jeremiah is a founder of the Web Application Security Consortium (WASC), and was named to InfoWorld’s Top 25 CTOs for 2007. He has authored dozens of articles and white papers, is credited with the discovery of many cutting-edge attack and defensive techniques and is a co-author of XSS Attacks: Cross Site Scripting Exploits and Defense.

Jeremiah will be delivering the opening keynote at the SANS AppSec Summit in Las Vegas on April 30. Here are his thoughts on application security.

1) How big is the AppSec problem?

Big, really big. The answer also depends on how you measure. Let’s start by looking only at websites, where I specialize, and the total number of them. Right now there are around 677 million on the Internet — growing at tens of millions per month. Obviously not all of those websites are important from a security perspective, but maybe those supporting SSL are, which is about 1.2 million.

If you are already familiar with application security, then you already know the vast majority of websites contain serious vulnerabilities, whether they support SSL or not. Presently, far more websites and source code is being deployed each month than is being reviewed for security. Web security will progress only when that divide is addressed and eventually changes.

While the precarious state of Web security has been known for some years, it has only been within the last few that the bad guys exploiting these issues have given urgency to the problem. The best way to characterize today’s threat is by quoting Verizon’s Data Breach Investigations Report (2012): “Web applications abound in many larger companies, and remain a popular (54% of breaches) and successful (39% of records) attack vector.”

If we expand the scope to all forms of software, which includes desktop, mobile, Web services, etc. Like I said, BIG.

2) The software community is made up of a lot of smart people. Why haven’t we been able to solve the problem of writing secure software?

It’s true. There are a lot of smart software security people out there, but no matter how many exist right now, more are needed — thousands — tens of thousands more will be needed.

To provide some context, through his BSIMM research of 35 large software security shops, Gary McGraw (CTO at Cigital) recommends roughly 2% of all programmers should be software security pros. With a world-wide population of 17 million programmers, eventually the industry will need 340,000 software security pros.

I do not believe anyone has a strong idea of what it takes to produce “secure software” and keep it secure while in a business production environment. At least, there’s little representative data that supports what security controls actually make a measurable impact on the security posture of an application. All the data available is largely limited and anecdotal. The smart software security people should be able to find a way to make acceptably secure software on a deadline and without breaking the bank, but a lot more research is going to be required.

3) Is the problem solvable? Is it really possible for developers to write secure software? If so, where should developers and businesses start? What are the first changes that they need to make?

If the definition of “secure software” is one in which no vulnerabilities exist, then no, the problem is not solvable. Software will always have bugs, and by extension, security vulnerabilities. This by no means makes the challenge a fools errand because perfect software is not the goal, nor does it need to be. If the definition of secure software, or secure-enough software, is one in which facilities business with an acceptable level of risk, then the problem is absolutely solvable. The challenge is outcome measurement. We’re talking about constantly measuring security postures and mapping those outcomes to security controls relative to the business value being generated.

This process can only be started with buy-in from the business stakeholders. Once security is valued and is required by the business, then for a development team, finding the best place to start becomes easy.