Regular spidering should be part of a web applications maintenance regiment. Of course, there are plenty of free and commercial tools to do it for you. Vulnerability scanners will typically come with a powerful spider function. On the other hand, public search engines like Google already do most of the work for you. In particular Google does provide you with a nice insider view into your web application. All you need to do is register. In order to register, you first need to sign up at http://www.google.com/webmasters/tools/ . Next, you need to show that the site is actually yours. You may do so by adding a special file to the site, or by adding a meta header with a specific code. The process usually only takes a couple minutes.
But why should you bother?
First of all, Google webmaster tools will provide you with more insight into how Google indexes your page. The section I am looking at first is “crawl errors”. It will tell you if any pages were not found, or if pages timed out. You can also check how robots.txt limited the crawl.
The “HTML Suggestions” section will tell you about search related errors in your HTML code. For example, inefficient use of META tags.
A common question we get at the Internet Storm Center is how to remove a URL from Google’s index. Webmaster tools will again help with that. The feature is a little bit hidden, but if you select “Site Configuration” and “Crawler Access”, you will see the “Remove URL” feature. This is probably the fastest way to remove a URL from Google.
Finally, the “Labs” section: One feature Google is currently experimenting with is malware detection. If Google found malware on your site, it will tell you about it. Sure, this is not the ideal way to find out about malware on your site. But better late then never.
I think Google’s webmaster tool is a “must have” for everybody running a website. While not strictly a security tool, it does help with security (and of course search engine optimization).
Posted by Jason Lam on October 15, 2009 – 1:42 am
Late 2008, Jeremiah Grossman and Robert Hansen publicized the clickjacking problem and got the web app security experts all trying to come up with solutions. One of the more viable solution is the X-FRAME-OPTIONS header that allow a site to control whether its content can be within a frame. There are two settings to this header, DENY blocks the content from being in a frame and SAMEORIGIN only allows the content to be framed by pages within the same origin. While it is not the end all and be all of clickjacking solution and won’t work in some sites that extensively use frames across multiple sites, it is considered as a more reasonable approach for sites to protect their own content.
Johannes Ullrich and I got a little curious about the adoption of X-FRAME-OPTIONS header in the major sites. We went and got the list of top sites according to Alexa and see if the top sites actually use X-FRAME-OPTIONS header. We expected the results to be low, but when we looked at the results from visiting all these sites, we were a slight bit surprised. Of the TOP 10,000 sites from the Alexa’s list, only 4 sites have the header specified. That number obviously is very low.
Not having the X-FRAME-OPTIONS header does not mean these site are not protected from clickjacking, maybe they are using some type of Javascript framebusting, we haven’t looked at how many have framebusting deployed yet. It is widely believe that the Javascript only defense is not as good as the attacker could disable Javascript in the frame by setting SECURITY=RESTRICTED on the IFrame to disable the Javascript defense.
As a side note, I am doing my personal part on promoting the X-FRAME-OPTION. In the next major update of my SANS DEV422 course (soon to be 522), there will be emphasis on various defenses against Clickjacking that a site owner can leverage.
Posted by Jason Lam on June 25, 2009 – 12:34 pm
I regularly get consulted on various web application security issues and defensive strategies. One of the recent “frequently asked questions” is around database encryption of web application. My answers to these kind of questions usually lead to awkward looking faces. I always start off asking more questions about the requirements, “Who are you trying to protect the data from?” and “What data are you trying to protect?” The answers to those questions are usually good indicator whether the person is on the right path or not.
In most cases, database encryption does not prevent the hacker from accessing the backend database via an application compromisse. The reasoning is very simple. If the web application needs to be able to access the data for normal operation and hackers are able to compromise the application, the hackers can essentially access the same data by controlling the applicaton (attacker owns the application). Encryption does not prevent a hacker to access the backend database if the hacker can control the application.
Some may argue that if the database encryption is per user based, then one user’s compromise does not lead to compromise of another user’s data. That is very valid but that usually requires authentication of the user to some backend pieces (or the database itself). Maybe authentication alone is sufficient for the security requirement? The database can provide access control based on user’s identity so one user’s compromise does not lead to access to other user’s data.
You might think I am an anti-encryption person. The fact is – I am not. Encryption totally has its place for web application security reasons. Asymmetric encryption usage to protect data that is “one way” in nature is a good example. As a retail store trying to store the user’s credit card details for recurring monthly payment, the data should be encrypted with asymmetric encryption so that the Web frontend can never read the data back but another party with another related key can. Also, there are numerous other scenario where encryption should be used, such as backup, protecting against rogue DB administrators and against physical theft of database machine.
I often recommend a threat risk assessment before deciding on the database encryption solution. That would allow you to quickly understand the threats that affect the web application and also possible countermeasures that can help protect against the threat. Due to the cost of deploying and maintaining database encryption, I am seeing very limited deployment of database encryption, most folks tend to turn to other alternatives for risk mitigation.
Encryption is a useful defensive technology that can help protect the web application but it is not a silver bullet, blindly deploying it may not address any threats that you are trying to mitigate. The managers and auditors tend to request these technologies to be deployed to defend against web application compromise after seeing another competitor getting compromised thru web application security flaws (eg. SQL injection). Encryption is likely not the best choice of defensive technology given the scenario.
