|
|
Abstract: This essay compares and contrasts security projects with an aim of identifying the differences.
With luck, this approach might give us a list of metrics that separate good projects from bad.
This essay is laid out firstly with some background and observations, and then a list of metrics that have been identified. These are described in summary form; my intention is to catalogue potential metrics and explore their effects, rather than to definitively promote a theory of metrics as the underpinnings of security.
In closing, the appendix includes a balance sheet of various projects measured informally against the metrics. This for many people will be the meat of the essay and many might like to skip right there and see how their favourite projects handle.
Caveats: this table could be dangerous to your health if used carelessly. Only an understanding of each metric is likely to lead to anything helpful; the numbers are meaningless by themselves, and the table is cute but likely misleading in uninformed hands. Simplistic copying of the techniques suggested would sink you to the lowest common denominator effect, a.k.a. "best practices", and should be avoided. Further, the numbers represent the personal opinions of this one author and have not been scientifically tested nor peer reviewed. Finally, the assessments are three years old and have not been updated. Much has changed in that three years.
This particular thread came out of the examination of a project that claimed an aspiration to security. Out of the swirling claims and counter-claims arose the question as to how to scientifically ascribe the security label to a project.
This essay assumes the following shameless belief:
"I know what is a security project and what is not."
How do I know? At first, I didn't know how I knew, and this essay is an attempt to answer that question.
With each metric I have aspired to cast it as a binary test. Does a project "have it" or not? Obviously, real life is not so accomodating, and sometimes we have to be softer in our assessment. Other times we get the test wrong, and have to reconsider. Does the metric have no merit? Is it trying to test two things that are better off broken apart?
All these and more challenge the simple binary view. Nothwithstanding, if we can achieve a sufficiently large set of metrics, and adequate quality within each one, then we can hope for an overall judgement to be meaningful.
Open source is widely touted as an import factor in security projects. Some see it as essential and others see it as dangerous. What is clear is that open source helps in some ways, and hinders in other ways. What is also becoming clearer as time goes on is that it is by no means the only criteria.
I personally do not buy the arguments of those against open source. Firstly, it is far more attractive to close off source for other reasons such as intellectual property and marketing reasons, and both of these agendas are happy to pronounce and promote motives of security under which they advance their cloaked agenda.
Secondly, although there is a direct benefit in closed source in that the attacker has a much higher cost of attack, there are many indirect costs to security:
As mentioned above, a closed security environment encourages closed policy and this eventually sucumbs to agendas that are counter to the interests in security.
As time goes on, policy and technical changes from within result in weakening of the defence, without the benefit of wider scrutiny; those interested in security generally have less resources than those interested in more profitable agendas.
Fewer eyeballs result in less absolute security, which tends to result in larger, more dramatic breaches.
A closed security environment almost always promotes a culture of perfect security. "Nothing to worry about, our security team is the best in the world." For this reason, when breaches do occur, corporations are unprepared. They behave badly and costs skyrocket for both the corporation and the user.
I suggest that the seduction of closed security is inevitable, even amongst the open world of the net. I therefore consider a closed security policy to be a demerit, and this strongly influences preference towards open source. Yet, this whole area is controversial, and a subject for future research.
If the source is readable, this presents a powerful incentive for the security project to do "the right thing." Being found out in source can result in substantial damage to reputation.
If the source is usable this provides a valuable force for security. Even if a project does not accept a security issue, individual users or even breakaway projects can fix the source and distribute a competing version.
Many times projects have been spawned on this basis, and perhaps the most famous are the chart leaders: OpenBSD split from FreeBSD on partly security grounds and while the event was traumatic, it could be suggested the new result brought better choice, better flexibility and ultimately better security for the users of both projects.
Does the user have a meaningful but compatible choice? Are there viable second sources of the technology, thus encouraging meaningful and critical debate on security issues?
There is a substantial difference in security behaviour between projects with competition and those without. In short, 'monopoly' projects do not easily accept security discussion, and other agendas and interests overrule the discussion. In contrast, the existance of a competing project makes it much harder to avoid security 'home truths'.
The existence of competition in security lends itself to a standard game theory analysis. As one player will be rewarded for focussing on security, and another player may be punished when they permit other agendas to suppress security, there is a direct benefit to competition. Yet, from game theory, there is also the opportunity for players to band together in a cartel presenting a limit on this benefit.
There is plenty of evidence of cartel-like behaviour in the security field, found in standards committees for Internet groups, best practices and regulatory groups for banks, etc. Even though the players may succeed in suppressing the security agenda through these means, the temptation to cartelise is more or less transparent, and competition remains a positive force.
Disclosure, like open source, provides both for a series of metrics that are identifiable and measurable, and also a view to an attitude to security. Does the company reveal problems? Or does it try for the cover-up?
Thankfully we are moving beyond the times when any security problem could be covered with a mixture of bluster and double-speak. "We listen to our customers and they do not tell us of these concerns..." Wide-spread security failures in the retail side of the Internet, perhaps most poignantly epitomised by the 2004 California 18xx law of disclosure and the resultant Choicepoint affair, have now made it unreasonable to engage the cover-up.
Still, nothing changes that fast and looking at metrics on how companies and projects reveal bad news is instructive.
First up, that old favourite of the press is Patches. Not how many, but whether. If a project issues a series of security patches, that is a positive indication of improving security. The alternate is no patches, which either indicates a static viewpoint on security, or a cover-up.
All security has weaknesses. No security is absolute. What then is left between the the unobtainable perfect security, and the practical delivery of a product? Weaknesses!
Now, your average user can look at weaknesses in several ways. She can learn and rectify the weaknesses in some higher layer. She can catalogue and compare the weaknesses between products, and select products with weaknesses that match her strengths. Or she can simply ignore the weaknesses, hoping that other factors will cover for the gap: brand, insurance, lucky protecting factors, unlucky attackers, confusion of goals, or future promotions.
For the latter approach, there is nothing we can do. But for the more discerning and responsible consumer, a product can list and develop its weaknesses, so that the user can make an informed choice. Such a product is more powerful, simply because the informed choice is now possible.
The issue here is the listing of some weaknesses, not all. If the product makes available no weaknesses, then it is simply stripping the user of choice and it is an inferior product. If however, the product lists some weakneses but not others, then it exposes itself to scrutiny. In time, all weaknesses will tend to come out under the attention of the market. So we limit the metric to a clear, available explanation of some weaknesses of the product, not whether these are in some sense complete or accurate.
The canonical example of weaknesses might again be the PKI/SSL products as against the SSH family. In the latter, there is an attack known as the man-in-the-middle, which is theoretically possible in the first instance, or whenever keys are rolled over by servers. This is acknowledged freely by the SSH community; all systems administrators know of it and work with it.
In contrast, the MITM is never talked about in the PKI community. According to their marketing, the MITM is eliminated by the use of CA-signed certificates. This is not the case, the MITM has simply moved. The PKI structure has shifted the burden from the users to the CA, and that latter is now the location for an MITM, either against it, or by it. The PKI community does not acknowledge this threat to users, and therefore deserves its demerit for hiding security problems from its customers.
An allied disclosure is the routine admission of mistakes. Some few security organisations will readily stand up and declare that such and such was a mistake, and there is a need to rectify that mistake. For example, the OpenSSH group recently declared their DSA "improvements" a mistake that was to be cleared out at some cost in future releases [3]
Damien Miller: Firstly, increasing the default size of DSA keys was a mistake (my mistake, corrected in the next release) because unmodified DSA is limited by a 160-bit subgroup and SHA-1 hash, obviating the most of the benefit of using a larger overall key length, and because we don't accept modified DSA variants with this restriction removed.
In contrast, most security vendors will not admit mistakes. They will twist and turn and do anything to avoid their customers thinking any negative thoughts about them. Obviously, the downside of hiding mistakes is a bit of a lottery, but consider how much easier it is to improve security in the face of a real mistake? If your policy is to hide the mistake, that makes it harder to improve security. If your policy is to declare the mistake and then move on, that's one less barrier against the real goal, which is what we turn to next.
Some projects have elevated security to a more serious position, that of a goal. The Goal of Security is a very serious statement that effects the entire structure and makeup of a project. A brief description of what a goal really is would help here, as most will think it is some corporate fad that is better off left in a Dilbert cartoon.
A goal can be defined as:
the one thing that if you have achieved it, you have done what you should have, and if you have not, then anything else that you might have done was in vain.
A goal is a statement that drives all decisions. We should be able to take any decision and walk back up the logic tree to the root of the goal. The decision should help us meet the goal, and if the decision goes against the goal, then it is a bad decision.
Why is this useful? Because life is complex. We are faced daily with complexities that can be seen from many different angles. Money, jobs, reputation, beliefs and other agendas depend on getting the outcome that we want from every problem, but this only works if everyone is aligned within these disparate forces.
Setting a clear goal to which we can all agree helps to clear the air and allow good decisions to be reached. It depersonalises the discussion, allowing us to concentrate on how we reach the goal, not who amongst us knows more, talks better or has the faith of those that matter.
If the goal is security, and the issue at hand is important, we should then be able to match the issue to the goal. If not, we have to ask ourselves whether we are ready to make that decision, or whether it is really important?
"FreeBSD has a security goal, but OpenBSD has *the* security goal,"
Jeroen van Gelderen [4].
In some senses, the goal of security is just that - security is our goal. How it effects the production of code and delivery of product to users goes beyond those simple words.
Issues such as reputation in the marketplace shrink in importance. Battles with competitors become less of an issue; although there remains the important caveat that security that isn't delivered does no good. Old assumptions can be examined against new threats. Business needs can be measured against improvements in the goal, do the clamours and cries of our users take us away from our goal or towards it?
While it is possible to have many goals, this leads to conflicts. Typically, we create limitations to our goal that reflect and relate other aspirations in a subordinated position. For example, our target market could be the security conscious system administrator or it could be anyone's Mom. A user interaction limitation could be secure out-of-the-box or it could require a WebTrust style audit, a degree in contract law, and a signed affidavit that a CPS has been read.
It's important to also declare that there is some substance here that needs to be understood. Many talk about security as if it was important, and slip in comments like "security for our users is our goal." But these are just words, and do not inform the projects as a goal does. If the words appear in a press article, but nowhere else, then we can be sure they are just borrowed advertising, and a red flag for real security.
In an influential paper, Eric Raymond introduced the Bazaar as an important institution behind the success of many open source projects [5]. Aside from suggesting some correlation with success, the image appeals to the Internet community, who are in general advocates of libertarian and free market traditions in their many guises.
Yet, successful security projects are not bazaar-like. They exhibit specific deviations, and one of those is the presence of a very high level director role. This role has significant power in overriding decisions made routinely in a bazaar context - release decisions, patch sets, importance of code sets, etc - which are all motivated by an attention to the overall security of the project.
The Security Czar interprets the Goal of Security. More subtly, the role exists to coordinate horizontally across the various teams that might form part of a project. In the commercial world, these teams are termed silos and the challenge is to bring the essential parts together without disturbing the ability of the independent teams to deliver. The security czar performs this role by overseeing the disparate interests, drawing together the parties, and having the power to push for cross-silo changes, all the while looking forward at the goal.
Code bases become very complex. Success is built from mad choices, insane designs and dollops if not smotherings of luck. Most all successful projects go through a crazy period of dramatic growth on poor quality code bases, and security projects do not appear to be an exception.
Getting rid of that code is generally implausible, but code audits help a lot. When projects back off from madly stuffing features in and start thinking about the security of their code in a more rigourous sense, we call this process a code audit.
Walking through the entire code base can take a long time. It could take one person a month to fully audit a thousand lines of difficult code, and the cost of that is probably the opportunity lost in another thousand lines that never get written.
Yet, consistently when these audits start up they signal a maturity phase in the project. It's serious! What's more, bugs start appearing; the number of patches, bug fix releases and security advisories climbs rapidly as problems are flushed out.
Perversely, a project with threats gets more plus points than a project without. There are these big issues with threats:
The threat should be a clearly identified and understood,
the threat should be aggressive and be active, causing damages and costs if not kept at bay,
the threat should be measured for frequency and cost.
A threat that meets these high challenges I call a validated threat. With a validated threat it is possible to predict with greater or lesser degrees of accuracy how much work and protection it is worth building in. In contrast, threats that do not subject themselves to validation also have a way of being blown out of proportion or ignored.
An important thread here is whether a security project faces validated threats, and thus has a strong enemy on which to focus and calulate, or whether its threats are unpredictable and thus of insufficient quality and quantity to guide decision making. Having a validated threat means a lot to a security project, it is the difference between economic security on the one hand, and on the other, waste and uncertainty in pursuit of this year's bogeyman.
The canonical example of threats is the never ending comparison between the SSL secure browsing and SSH secure terminal projects.
In SSL the threat was not taken from experience but was taken from cryptological theory stretching back to military and intelligence traditions. In fact, the threat was circularly created from the known threats that could be covered, rather than the threats that existed for the target user base [6]. In contrast, SSH evolved to solve a known problem - password sniffers. With this clear and validated threat in mind, the security model created was then aligned and sufficient to defeat the threat.
SSH defeated its threat dead. So much so that SSH saw no active and aggressive threat for a decade [7] This leads to another issue, one of more strategic subtlety. In the absence of a current, active and aggressive threat, it is very difficult for a project to advance.
In contrast, SSL today faces an active and aggressive threat - that of spoofing or phishing. More properly, the flagship application on SSL, secure browsing, is beset by security failures and losses to users variously estimated in the order of billions per year. It is this very threat which provides the incisiveness that helps the project architects to focus and channel their energies to what matters.
Perhaps surprisingly then these two great projects earn a point each. SSH was strongly aligned and built to a real threat, but it so killed that threat that it now faces no direction and thus has trouble advancing. This is a problem, but it falls in that class known as "a nice problem to have." In contrast, SSL derived from muddled threat origins, and only recently has had to face up to its attackers. This gives it the focus and direction to advance in security, although of course there is no guarantee that the signpost will be noticed.
A higher order of active threats exists. It is not enough to simply identify that threats exist. It is important to quantify how important each threat is. Ideally, we want to get to the point where we know a likelihood of attack and an estimated cost of the attack. This point often remains elusive, because the attacker is aggressive and will not be boxed into our statistical models.
Yet it remains an ideal if only because a large body of science deals with risks in this fashion. Think of the insurance industry, which commonly says that if it cannot be measured, it cannot be insured. It is with these measurements that we take a security project into the next domain: understanding the tradeoffs between protection and vulnerability so as to best apply our limited resources where they do most benefit.
Few security risks are measured. Of those that are, there are great suspicions as to their value, especially from those that may be criticised by the measurements [8]. So in considering the existence and the adequacy of security measurements, we err on the side of existence, and accept even controversial measurements. Hopefully, one day these controversies will resolve themselves and we can be more critical, but for now, simply getting to the point of measurement is worth a point.
For many, security means cryptography. But cryptography is only a small part of security, indeed Internet security itself is a field with as much in common with physical security, national security and stock prices as cryptography.
Yet, for its small part in code lines, it has had an over-bearing effect on real security. Here's two ways in which we can measure judicious use of the cryptography weapon in the battle for user security.
Does a project dare to go against popular wisdom? It would seem that at the root of many projects' failure to address security developments is slavish devotion to security models, standards and RFCs that are well documented, popularly standardised, but somewhat or totally useless for Internet security.
PKI is the canonical example of religiously blind devotion to a model. Invented in the 1970s as a way for the postal agencies to do electronic mail in a world of store-and-forward, it has never managed to cross-over in any academically sincere sense to the immediacy of the Internet. Yet large portions of the net devote their security to that model, declining to recognise the inapplicability in the roots of design and architecture.
In contrast, Skype, the VOIP company, ignored most all designs and schlapped on some home-built crypto onto their free voice telephone application. It got them up and going! The important issue was that the crypto gave a fairly good protection for something that needs no more than fairly good protection - phone calls between net friends. Their rebelliousness brought the scorn of the professional cryptographic community down on their heads, but their users loved them. Skype stand as the company that has done most to bring the benefits of cryptography to the masses, primarily because they rebelled.
So who cares enough about security to ditch standards when they get in the way? A hard question to answer, as asking it is like using lit matches to search for a gas leak.
One question that demarcates those that are strong on security is whether the delivery includes full-on strong crypto or whether it wimps out on some excuse or other. Most all operating systems wimp out as long time victims of the United States' war on security. Cryptography export regulations have dampened the appetite of many software projects to incorporate good security protocols, and had a chilling effect on students of computer science. Better not study that stuff - it's too hard to get a job doing it. Even though today's climate is a slightly easier one than in the 1990s, regulation continues, memories are long, and new restrictions are just a 9/11 away.
What this means is that cryptography is simply not used if one can avoid it. Unix suffered a decade of difficulties in export because the DES algorithm used in the password protection wasn't exportable. The result was less security - the OS was shipped without protection to overseas markets, and even local markets suffered much higher transaction costs due to the "domestic" and "international" version confusion.
We can also see this in the failure to use simple tools where they would help. Only in recent years is it becoming routine to use a message digest to create a signature; a decade back that would have been seen as "scary." Yet, a message digest is one of the easiest tools to use - it takes one input and puts out one output. Even though it is far harder to write a reliable protocol than it is to use a message digest, programmers think nothing of doing their own client-server protocol but rarely add in the basic tools of cryptography.
These are some of the things that have no bearing on whether a project makes the grade or not:
Number of bug fixes,
Number of security advisories,
Size of security team,
Number of times security is mentioned by talking heads at conferences or in the popular computing press,
Number of pages on the web site that talk about security.
How many times the code has been rewritten.
Confusingly, some of these things may have negative correlation with security such as the talking heads and the patch count. But that's not worth relying upon.
The above approach is definately an individual list of choices. The main intent of this essay is to spark a debate as to what we can learn from other projects, and whether these observations can be useful in our own project.
There is relatively little sharing at the meta-level of security. The Internet is like a group of fiefdoms, with some projects apparently at the throats of other projects, when to an outside observer the end result is indistinguishable. How else is one to interpret the sometimes dismissive comments seen between SSL and SSH, the BSDs and Linux, between Microsoft and everyone else?
All of the easy wins in security can be considered gained. We can assume that the easy things are being done, which leaves the difficult, the subtle, the complex. To see how these issues in security unfold, projects need to look beyond their own fiefdoms, and no better example exists than that of a similar project dealing with a similar threat.
Below is presented a 'balance sheet' of security metrics, following the discussion presented above. In order to interpret this table, refer to the meanings of each column in the main document, and also to the notes below.
The choice of projects was one of the biggest difficulties. In the end, I settled on the existence of a substantial and identifiable code base, substantial product(s), a brand name of some potency and an identifiable community surrounding these elements.
There is of course some cross-over; threats to one project are threats to another, and software is often shared. Microsoft is too broad in the project sense for this measure, but we are measuring as much for the attitude of the company as any individual project, and Microsoft is famously homogenous. Not including it makes the table unrealistic. Others of the projects are small and unknown, but were included because I had enough information to make a judgement.
Each of the selections are done with only scant research. For example, I was not able to find any writings that indicated FreeBSD has a goal of security; this does not mean it is unformalised, but not being able to find it achieves the same effect.
The ordering is loosely from most points to least points, with one exception. Although the sheet is presented ordered by points, the points are only vaguely important. Indeed, my own project which is the exception to this ordering lacks points, and I am not worried for the simple reason that many of the metrics are not applicable as yet.
The object of the exercise is not to get as many points as possible, but to deliver security to users. In this sense, many of the projects do far better than the metrics indicate; my intention here is to look at the way forward, not to applaud the journey so far.
Following points is also to follow Best Practices. Likewise, to adopt this list as Best Practices would be to abrogate ones security thought to a committee, one that has no responsibility for securing your project. The issue with these suggestions is to apply that which works for you, and to do that means you must understand how these suggestions work in your context.
Analysing our own work is of course subject to more than the normal conflicts and biases, and for that reason Ricardo is taken out of order and placed as the last line. It is primarily useful as a thought exercise for us. It is useful to be fair and tough and to test the criticisms on ones own work. Indeed, if the medicine offers nothing to ourselves, how can it be of use to others?
Ricardo's relatively fair showing reflects its lack of user base; given few users, metrics like audit projects and patching make little sense and we prefer to spend our limited resources on new features and rewrites of old features. Also a note on the openness - even though the source is 90% open (some server code is currently closed) it lacks an open rating simply because the only people working on the code base are ourselves.
[1] Chipper, a Dutch smartcard money project, since folded into another project.
[2] The browser security model: all browser manufacturers operated what was effectively a closed security process. Although these remarks are backed by exposure to one manufacturer, there is no evidence to suggest any other browser is any more open, and plenty to suggest that the one I looked closely at is the most open. In the face of phishing no effective response was possible, and all browser manufacturers eventually adopted a closed security group in common for defensive motives.
[3] Federico Biancuzzi, OpenSSH cutting edge, SecurityFocus, 2005-12-19. Also see A new security metric? FC blog, 2005-12-24.
[4] Private conversation, sometime in the last millenium.
[5] Eric Raymond The Cathedral and the Bazaar
[6] Ian Grigg What's your threat model? Internet essay.
[7] Recently, automated username/password probing started up with a vengeance.
[8] BSD - The World's Safest OS, reporting on Mi2g's reports.