nick p on better security evaluations v3

Security Requirements, Design, and Evaluation: A Look at Now with Proposed Improvements

I put my proprietary development framework on Schneier's blog for free in 2013 in a reply to another commenter. There was hardly any demand for effective, ground-up security that I've specialized. Might as well be altruistic. Cleaned up version at the link below:

http://pastebin.com/y3PufJ0V

Then, the Snowden leaks happened. I was glad to see my framework addressed about every NSA attack vector, including in TAO catalog. Exception was physical implants but a rule I always preached is it's compromised if enemy got their hands on it. Far as source of my framework, what a TLA does becomes obvious if a person looks at previous vulnerabilities, successful mitigations, the root causes of failures, each layer/component in a system, and each type of interaction among them. Thinking along all these lines can help you find flaws or new categories of vulnerabilities in a straight-forward way. You get better odds at spotting the harder ones, too.

In any case, security certification isn't like most others. The current situation is that there are numerous standards, best practices, policies, guides, etc. Look at all, there's commonalities and also differences. You could say it's fragmented, inconsistent, redundant, and sometimes political. Too many standards, too: Common Criteria, DOD C&A, CIA's type, FISMA, and so on. Most people making them have never designed a secure system either. If you doubt the inadequacy, compare the technical components of *any* security evaluation to my framework. You'll see many missing requirements or mitigations. You might have also found it strange pre-Snowden that NSA's promoted EAL4-grade stuff NetTop or cross-domain solutions on Linux not meeting those requirements. Strange since NSA's evaluation criteria is where I got a bunch of them. Reasoning was more obvious post-Snowden. ;)

So, what can we do? Well, let's start by looking at what the Common Criteria (main standard) is doing. Current CC practice is to create Protection Profiles (PP's) for a given type of system (eg firewall), device (eg printer), or software (eg OS). It covers the security threats, protections needed, and minimal level of assurance. CC uses so-called Evaluated Assurance Level's (EAL's) 1-7 to rate the strength of the security claim based on what methods were using to catch vulnerabilities. The higher the number the more believable the claim. EAL1-3 is garbage, EAL4-EAL4+ is typical for big companies, EAL6-7 is highly secure stuff, and EAL5-EAL5+ is medium (typical + limited application of strong security).

An independent, private lab evaluates the product against the requirements with lots of paperwork. They're ridiculously expensive with evaluations taking long enough to affect time-to-market. If it's EAL5-7, local intelligence service gets involved, often gets the code itself, and pentests the product plenty before allowing a certification. Vast majority of products are under EAL4. That includes big vendors such as Microsoft and Oracle whose stuff gets hacked plenty. Smartcards dominate EAL5-6 evaluations. Finally, you can count on one hand the number of general purpose systems certified at EAL5-7. So, by government's own standards, almost everything out there is insecure as can be. At least it's certified to be, though. That it's garbage is right there on the certificate! :)

I'll briefly mention the Central Intelligence Agency's classification. They improve on the other scheme by doing something simpler a three-part scheme: confidentiality, integrity, and availability. Each attribute gets a rating from Low to High. So, a prospective buyer that knows CIA evaluated it can look at the C.I.A. ratings to have a quick idea of how good protections are for each attribute. However, CC is nice in that it lists each threat and countermeasure in the system. That gives more technical people a thorough idea of its strengths and weaknesses. Both types of measurement have their value to different kinds of buyers, though.

Where to go from here is pretty open to discussion. My proposal is to do evaluation and security description the same way I do my security framework: layer by layer, component by component, interaction by interaction, and with environmental assumptions stated. Mainly because it works that way & didn't most other ways. ;) An EAL6+ OS like INTEGRITY-178B, first certified to high-assurance under Common Criteria, running on a regular board won't be secure. Attackers may smash firmware, peripherals, etc. A more honest evaluation for something like Dell's INTEGIRTY-178B-based desktop would list (for example): processor (low), BIOS (low), peripheral code (low), kernel (high), key drivers (maybe high), disk/networking middleware (unevaluated), virtualization system (unevaluated), and so on.

The features of each would also be specified. Optionally, the level of rigor is stated on a per feature basis. This allows the vendor to incrementally develop both the product and assurance of each component plus clearly communicate the strengths or risks to the user. The person interpreting it would be a technical professional who would translate it to layperson speak for the decision-makers. The features, threats, and so on mentioned would also be adjusted in a reasonable period of time when new threats appear. Protection profiles would still be the norm to provide decent defaults without mentally overwhelming buyers. They just get adjusted periodically.

This means almost everything on the market, proprietary or open, is insecure against a motivated attacker. I'm almost for going further to create two categories at system level: Low and High. If it's not proven High security, then it's Low by default. Combining components means overall security is rated at level of weakest component. This should force vendors wanting to claim High security to invest heavily in bottom-up secure architectures for their product. Gives me hope to know some already did. As threats in my framework are added, the number of attack vectors and level of rigor of such bottom-up approaches would only increase.

Bottom line: Our evaluation process needs to be streamlined and the incentives changed to force anything declaring itself to have strong security to actually have strong security. It also needs an abstract rating for a quick glance evaluation along with a thorough treatment of strengths and weaknesses for a technical evaluation. I'd also recommend eliminating unnecessary costs and any incentives to loophole through requirements. Having mutually suspicious parties evaluate the security claims, while signing hash of resulting system image, might also reduce subversion concerns.

Evaluators should also work hand in hand with developers so they continuously have a clear understanding of the system with feedback going both ways when it's needed most. This will also benefit time to market since the evaluation is constantly happening by the independent QA/security team. The mechanisms used in High-security should be designed to benefit commercial sector as easily as government to maximize potential ROI for those developing robust solutions. Finally, governments involved in CC should each make a rule that only High security products are purchased for anything security-critical. That rule might give us half a dozen in each category from as many vendors within a year or two*. ;)

* Note: This is exactly what happened when they did this mandate in Orange Book days. Putting a profit motive on High security is a proven way to make it happen. The solutions developed were also diverse and extendable, too.

Nick P
Security Engineer/Researcher
(High assurance focus)