The System Defense Stack

System Defense Stack


I have scoured the Internet far and wide and found nothing similar to what is in my head regarding this topic, so I thought I’d provide the above as my conceptual map for system defense (not incident detection and response, which would be a separate stack, and of which many models already exist, often based off of an inversion of the well-known Cyber Kill Chain).  If I am duplicating effort here, please point me towards other models which already account for this material.  I am very interested.

The System Defense Stack is an abstract, layered approach to understanding defensive measures which must be breached or circumvented roughly from the top down in order to compromise a system.  Not every situation includes measures which fall into every layer of the stack, so this is a useful map for analyzing and identifying vulnerabilities in security measures at a variety of levels (individual processes, individual systems, networks, entire information technology architecture models, etc.).

A Simple Example

Let’s take CVE-2014-6271 (Shellshock) for an example.   In particular, we will assume the target system is a vulnerable server running httpd intended for use by the public.  The attacker will attempt to exploit this system’s vulnerability to cause it to present the attacker with a remote, interactive shell through which the system may be controlled.  The exploit is straightforward – the attacker leverages the fact that httpd launches bash instances using user-provided environment variables (such as header information).  If the attacking user simply provides a header crafted with the exploitative substring at the beginning of the header (() { ignored;};), he can cause the httpd process to attempt to execute the arbitrary code included after the exploitative substring (bin/bash -c ‘/bin/rm -f /tmp/f; /usr/bin/mkfifo /tmp/f;cat /tmp/f | /bin/sh -i 2>&1 | nc -l 1234 > /tmp/f’).

Notice that the exploit code first acts to remove any file named /tmp/f, then creates a named pipe called /tmp/f and outputs its contents to the standard input of an interactive shell session.  The output of that interactive shell is directed back to the named pipe which is configured through the nc command to receive input from port 1234 on the local host.

In terms of the stack, the attacker first had to establish access to the server, which was easy since it’s a public Internet system.  The process being hijacked by the attacker is intended for public use, so there is no authentication layer of protection here (anonymous internet users with access to the server can use this service).  Since the attacker is commandeering an httpd process to execute this code, the authorization layer consists of the UNIX user and group context of the httpd process and the file mode architecture of the system as a whole (which, by default, permit access to read, write, and execute the contents of /tmp for any user context, thereby making possible the file removal and creation of the named pipe operations).

If the attacker does gain proper authentication and authorization, execution control may still prevent his actions.  In our example, however, this is unlikely since the process (bash) being hijacked already has authority to execute and /bin/sh is surely authorized as executable on the system as well.  If the attacker were relying on the execution of his own software from bash, good execution control might prevent it.

This is pretty typical for a server of the theoretical type we have specified thus far, but we now have the ability to impede the attacker’s progress with mandatory access control, which might stipulate that this particular process to which the hacker has access is not permitted to write to or read from /tmp, despite that directory’s universally permissive file mode.  If that is the case, the attack has been stopped at layer four of the Software Defense Stack.

If the httpd process is operating within a well-designed cgroup (as part of a container, for example), the process control layer may provide some protection by restricting the httpd process’s access to the system.  There are no memory protection measures which might prevent the attacker from conducting the exploitative activity described above since the attacker is executing perfectly legitimate code in an illegitimate fashion.

Data encryption may be relied upon as a last resort if, somehow, the user context (authorization layer) of the httpd process, its mandatory access control configuration, and any additional process controls permit it to access sensitive information through the shell instance that has now been spawned under its authority.  Auditing of the system’s processes will collect information necessary to understand what has happened to, or what has been attempted against, the system.

Of course, establishing this remote shell is only the first step to what is likely to be a multi-step attack.  Once the attacker has access to the shell, (s)he will likely want to do something with the system, be it attempting exfiltration of important data or commandeering the system as a node in a botnet, who knows.  But, whatever the next step turns out to be, the same stack applies.  Given the great utility value offered to the attacker by the /tmp directory’s permissive file mode, one might scrutinize the /tmp directory alone with the System Defense Stack.  Knowing that access, authentication, and authorization have already been breached, the stack draws attention to mandatory access control or execution control which are likely to be the most fruitful means of securing the /tmp directory against misuse.  Establishing some execution control by mounting a separate file system with the noexec option on /tmp is a great, effective idea.

Using the stack, one can see through a quick analysis that a very well-designed public-facing httpd system will mitigate CVE-2014-6271 primarily through a combination of authorization (the UID of the commandeered httpd process), mandatory access control (the SE Linux context arrangement of the httpd process and files on the system), execution control (mounting /tmp with noexec so that the attacker cannot use /tmp as a platform for code execution), and finally, data encryption and auditing.


As readers of this blog may know (or infer from the blue hat reference), I am very interested in information technology architecture and system defense.  It seems to me that the current IT climate is becoming increasingly dismissive of “traditional system defense,” as it were, focusing instead on incident response.  It should go without saying that both are integral components of secure system architecture, but it does need to be said.  Preventing the occurrence of incidents is at least equally as important as detecting and responding to those which do occur.  Without prevention, detection and response is overwhelmed, and without detection and response, any weakness in prevention will be exploited with impunity.

To give an example of the kinds of pressures acting to displace focus from preventative defense to incident response, allow me to direct your attention to FireEye (an organization of great and indisputable value, despite the flaws I will cover here) who released a report entitled “Cybersecurity’s Maginot Line:  A Real-World Assessment of the Defense-in-Depth Model”.  Here’s their investors brief about the report which speaks very pertinently to my point here.  If you even superficially examine either of those links, you will likely notice the alarming bullet points:

  • Nearly all (97 percent) organizations had been breached, meaning at least one attacker had bypassed all layers of their security architecture.
  • More than a fourth (27 percent) of all organizations experienced events known to be consistent with tools and tactics used by advanced persistent threat (APT) actors.
  • Three-fourths of organizations had active command-and-control communications, indicating that attackers had control of the breached systems and were possibly already receiving data from them.
  • Even after an organization was breached, attackers continued to attempt to compromise the typical organization more than once per week (1.6 times) on average.
  • On average, attackers’ software exploits and malware downloads bypassed other security layers 1.51 and 122 times, respectively.

If you read the report, it’s even more dire, with claims such as the following littering the document:

“The implication is clear: no corner of the world is remote enough to avoid falling into attackers’ crosshairs, and current defenses are stopping virtually none of them.”

Of course, the presentation of this data is obviously tuned for marketing use.  If taken as true without further scrutiny, one is left to conclude that nothing other than FireEye (or a comparable product, of which, of course, there is none) can save you from inevitable destruction.  These claims are designed to maximize alarm, and so an inquiring mind is likely to desire further details.  Unfortunately, I read the report and it leaves unanswered the questions I consider most important.  How did these attackers breach all layers of modern security architectures?  Was it through misconfiguration or poor implementation within said architectures?  Were the particular architectures themselves flawed and vulnerable?  Or, more worryingly, was it an extremely adept feat of hacking that managed to exploit vulnerabilities in every well-implemented layer of the System Defense Stack?

If the answer is the latter, and it is true that even thoroughly well-designed and implemented defensive measures still fail to prevent their utter circumvention or destruction over 97% of six-month periods, then my friends, we are utterly boned.  I recognize that many, many more highly qualified people than I seem to believe this roughly to be the case, but I remain in the minority, and I am convinced that our current technological platforms are not so poorly designed as to allow for such a high level of regular compromise.  In my experience, every security breach I have witnessed could have been stopped by preventative measures as described by the stack I put forth above.  The biggest contributive problem relevant to these breaches was that these measures were either missing or poorly implemented.

My opinion, in short, is that the problems with information technology security are almost entirely related to negligence or incompetence (or, in rarer cases, lack of resources) on behalf of information technology professionals.

The report from FireEye lists its major conclusions regarding the details of the breaches they detected among 1217 FireEye system deployments in 63 countries across 20 industries as follows:

  • 208,184 total malware package downloads occurred, including 124,289 unique variants
  • 10,149,477 CnC transmissions to 35,415 unique CnC infrastructures

Knowing this, my immediate questions are:

  1. How was the malware executed on the target system?  One obvious method of counteracting such malicious software is to explicitly permit only trusted software to be executed on systems in the organization (software whitelisting).  I presume this measure was not taken in these organizations (and it rarely is, because it’s hard and IT staff are generally not sufficiently talented to implement it without obstructing organizational workflows to an unacceptable degree).
  2. How did communication from the compromised systems reach the CnC infrastructures?  Were basic outbound firewall rules in place, or were they lacking entirely?  If they were in place, were they poorly designed or were they simply necessarily broad and inclusive?

These questions are never addressed, though the report includes a little table where they purportedly “[peel] the onion, layer by layer” (sorry for the ugly formatting):


Customers That Reporting Using This Security Measure

Breach Rate
Firewall 212 100%
IDS/IPS 119 100%
Web proxy 138 100%
Network anti-virus 75 100%
Endpoint AV 169 100%
Other anti-malware 33 100%

Some strange discrepancies between the numbers in this report are observed.  The report doesn’t seem to indicate what subset of the organizations bearing the FireEye implementations are represented by this table, but it doesn’t seem possible that all 1216 (one organization implemented two FireEye solutions) organizations are accounted for here lest we be expected to believe that only 212 of 1216 organizations use firewalls (which would seriously and alarmingly further my point to a rather unbelievable degree).

Again, the data is inadequate for determining whether these breaches are occurring because of organizational deficiency or true inadequacy on behalf of the described security measures.  Though the report purports to demonstrate the inadequacy of the “current defense-in-depth model” (i.e. everything other than FireEye), it is unclear that it doesn’t simply reveal the myriad shortcomings of current IT professionals.  Furthermore, essential layers of the System Defense Stack are omitted entirely here; from the perspective of any given server, the elements of the table seem to correspond entirely to access (firewall, web proxy, IPS, network anti-virus) and slight execution control (Endpoint AV, Other anti-malware).  If we’re discussing only issues of actual malicious code being executed (and not process hijacking through buffer overflows or whatnot), then one single unrepresented layer of the System Defense Stack could have stopped it all:  execution control through software whitelisting.

But it’s hard.  On average, IT professionals aren’t very good, to be honest, and they just don’t try to implement such robust solutions, forgoing them as infeasible.  In most modern organizations, gaining access and authority over a system is typically as easy as a Facebook-themed spearphishing attempt, and that may well be all that is needed for the hacker to access sensitive data on poorly secured systems.


Defensive warfare is the noble pursuit of the righteous in a dangerous world.  We must stand as towers among men, deflecting aggression and enabling freedom.  The attackers have a far easier job, but if we are diligent, disciplined, and learned, we will demonstrate the superiority of virtue over vice.  Hopefully, this stack gives other information technology architects a frame within which to think about options for mitigating risk, and it makes more clear the value of doing so.

This entry was posted in Information Technology and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s