Scott Shapiro's 2023 book's aim is to answer three questions: (1) why is the Internet (still) so insecure? (2) how do hackers do what they do? and (3) what can be done about it? He recounts some historical events, the "five extraordinary hacks" of the subtitle, to tell the story, and introduces the terms "upcode" and "downcode" as the core concepts in his framework for understanding--where "downcode" means actual, implemented computer code and "upcode" means the social, political, and institutional forces providing incentives and governance. This is essentially a simplified version of Lawrence Lessig's four forces of law, social norms, markets, and code spelled out in his 1999 book, Code: And Other Laws of Cyberspace, and also reminded me of the framework in Bruce Schneier's 2012 book, Liars and Outliers: Enabling the Trust That Society Needs to Thrive, where the four forces are moral pressures (internalized incentives), social pressures (social/cultural incentives from other people), institutional guidelines and rules (formal rules, regulations, and laws), and security systems (locks, police, firewalls, fraud detection, etc. -- actual operational controls which may be implemented physically, in code, or by policies and practices). For Shapiro, Lessig's first three forces are "upcode" and only code is "downcode," and Schneier's first three forces and parts of his fourth are "upcode."
I found Shapiro's answers to the first and last questions of his project fairly unsatisfying by comparison to Schneier's book (or to, say, Ross Anderson's massive Security Engineering book, which you can find free online and which is the number one book I recommend to people interested in computer security). His upcode/downcode framework is, I think, too simplistic, and while he correctly identifies some of the specific factors relevant to his five extraordinary hacks, he also makes some bad arguments and gets some things quite wrong, and I think misses some of the key issues that have become highly relevant today with the rise of large language models (LLMs).
His answers to the second question, specific to the hacks he describes, however, are generally quite good and detailed, and this is where I think the book is at its strongest. The five hacks are:
(1) The Robert Morris Jr. worm. (2) Dark Avenger's viruses. (3) The breach of Paris Hilton's T-Mobile Sidekick. (4) The Russian GRU (CrowdStrike name: "Fancy Bear") breach of the DNC and DCCC. (5) The DDoS attacks of the Mirai botnet.
Each hack is presented over multiple chapters, with historical context and explanation of technical details, as well as relevant social and psychological (and legal) factors, in his upcode/downcode framework. I share a complaint with other reviews that Shapiro has a tendency to invent unnecessary terminology ("vorms" for a virus/worm hybrid, and "mudges" for malicious nudges, of the sort commonly referred to as "dark patterns" or that are examined in depth in Natasha Schüll's 2012 book, Addiction by Design: Machine Gambling in Las Vegas).
My biggest factual complaints with Shapiro come in some of the background context he supplies to the 1988 Morris worm, when he gives a brief history of the UNIX operating system that begins with its predecessor, Multics. Shapiro notes that his father, like Morris's, worked at Bell Labs. (And disclosure: I used Multics beginning around 1978-79 and as a Multics developer from 1983-1989.) Bell Labs is the likely source of the issues in Shapiro's discussion of Multics. Shapiro describes the history of Multics like this (p. 26):
A large team from MIT, Bell Labs, and General Electric, therefore, decided to develop a complete multiuser operating system as a replacement for batch processing. They called it Multics, for Multiplexed Information and Computing Service.
The Multics team designed its time-sharing with security in mind. Multics pioneered many security controls still in use today--one of which was storing passwords in garbled form so that users couldn't repeat Allan Scherr's simple trick. After six years of development, Multics was released in 1969.
The military saw potential in Multics. Instead of buying separate computers to handle unclassified, classified, secret, and top-secret information, the Pentagon could buy one and configure the operating system so that users could only access information for which they had clearance. The military estimated that it would save $100 million by switching to time-sharing.
Before the air force purchased Multics, they tested it. The test was a disaster. It took thirty minutes to figure out how to hack into Multics, and another two hours to write a program to do it. "A malicious user can penetrate the system at will with relatively minimal effort," the evaluation concluded.
The research community did not love Multics either. Less concerned with its bad security, computer scientists were unhappy with its design. Multics was complicated and bloated--a typical result of decision by committee. In 1969, part of the Multics group broke away and started over. This new team, led by Dennis Ritchie and Ken Thompson, operated out of an attic at Bell Labs using a spare PDP-7, a "minicomputer" built by the Digital Equipment Corporation (DEC) that cost ten times less than an IBM mainframe.
The Bell Labs team had learned the lesson of Multics' failure: Keep it simple, stupid.
Based on this description, one might infer that the US Air Force did not purchase Multics and that it faded away, but in fact they purchased multiple Multics systems and operated them for decades. One might also infer that the Paul Karger and Roger Schell-led penetration test was the end of the story, when it was just the beginning--the issues identified were corrected, and on September 1, 1985, Multics was the first operating system to receive a B2 security rating from the National Computer Security Center (a project I participated in). Shapiro's "thirty minutes to figure out how to hack" and "another two hours to write a program to do it" is referring to the "Master Mode Transfer" vulnerability in table 3 of the Karger and Schell report, and it should be noted that these time estimates are for a computer science graduate of a major university with moderate knowledge of Multics design and nine months experience as a Multics programmer with access to the source code (which Multics provided to its customers for all except the "unbundled" extra-cost software). Shapiro's "A malicious user can penetrate the system at will with relatively minimal effort" is an accurate quote from the report, but not from its conclusion, rather, from an earlier section titled "Multics as a Base for a Secure System" that is referencing the prior section, "Multics is not Now Secure." The next section says that "In the long term, it is felt that Multics can be developed into an open secure multi-level system by restructuring the operating system to include a security kernel." And that was, in fact, done -- the B2 security rating was earned, in part, for demonstration of the security of the Multics Trusted Computing Base (TCB), aka, the "kernel."
Shapiro's description says that Multics was "complicated and bloated." While it is true that some joked that Multics stood for "many unnecessary large tables in core [memory] simultaneously," it was incredibly small and streamlined by modern standards. Karger and Schell's followup paper, "Thirty Years Later: Lessons from the Multics Security Evaluation," notes that the 1973 Multics TCB was 628K bytes, and that the security policy alone of SELinux is 2.5 times larger, not even counting the Linux kernel itself.
It is also worth noting that while Shapiro's description suggests that the Bell Labs team broke away and started afresh after this failure, Bell Labs actually withdrew from Project MAC very early in the project, in March 1969, while the Air Force assessment took place from July 1972 to October 1973. Ken Thompson has admitted in an interview that he's never used a Multics system.
It is undeniably true that UNIX outcompeted Multics, but the reasons for this are many and include that Honeywell had three mainframe operating systems, GCOS (formerly GECOS), Multics, and CP-V (and its successor CP-6), and GCOS was the one that was given the most support, development, and marketing resources. Unlike UNIX, which became open source in the early nineties and its essentials cloned as the open source Linux, Multics was not made open source until November 2007 and never ported to alternative hardware. I highly recommend that anyone interested in operating system history and computer security spend some time perusing Tom Van Vleck's Multicians website, as there are still lessons to be learned from and about Multics. (And you can now run one yourself on a laptop or even a Raspberry Pi, via Honeywell DPS8 hardware emulation.)
A few pages later (pp. 29-30), Shapiro writes again about Multics, and rightly gives credit to the Karger and Schell report for something UNIX co-creator Ken Thompson regularly is given credit for coming up with:
When Ken Thompson won the Turing Lifetime Achievement Award in 1984, the highest honor in the computer-science community, for developing UNIX, he devoted his lecture to cybersecurity, a first for the Turing lecture. In the first half of his lecture, Thompson describes a devious hack first used by air force testers when they penetrated the Multics system in 1974. They showed how to insert an undetectable "backdoor" in Multics. ... The moral Thompson drew was bracing: the "only program you can truly trust is the one you wrote yourself."
Thompson's lecture, published as "Reflections on Trusting Trust," ends with an acknowledgement in footnote four that he didn't come up with this idea, citing "Unknown Air Force Document" (which is the first Karger and Schell document linked above). The specific idea is not just a trapdoor or trojan horse, but a trapdoor in the compiled code that is not present in the source code, but is inserted by the compiler itself -- including that insertion functionality itself being automatically inserted into the compiler when the compiler itself is recompiled. The idea is that the trapdoor cannot be observed in the source code, it's not present there, but is only in the compiled binary. Karger and Schell describe it like this (in section 3.4.5, "Trap Door Insertion"):
Clearly when a trap door is inserted, it must be well
hidden to avoid detection by system maintenance personnel. Trap doors can best be hidden in changes to the binary
code of a compiled routine. Such a change is completely
invisible on system listings and can be detected only by
comparing bit by bit the object code and the compiler listing. However, object code trap doors are vulnerable to
recompilations of the module in question. ...
It was noted above that while object code trap doors
are invisible, they are vulnerable to recompilations. The
compiler (or assembler) trap door is inserted to permit
object code trap doors to survive even a complete recompilation of the entire system In Multics, most of the ring
0 supervisor is written in PL/I. A penetrator could insert a
trap door in the PL/I compiler to note when it is compiling a ring 0 module. Then the compiler would insert an
object code trap door in the ring 0 module without listing
the code in the listing. Since the PL/I compiler is itself
written in PL/I, the trap door can maintain itself, even
when the compiler is recompiled.
An object code compiler trap door, therefore, would mean that you can't even trust a program you wrote yourself! There is, however, a route to addressing this problem, which was the subject of David A. Wheeler's doctoral dissertation in 2009, "Fully Countering Trusting Trust Through Diverse Double-Compiling." (Note that Karger and Schell did demonstrate an object code trap door in the Multics penetration test, but not a compiler object code trap door.)
Shapiro does mention the NCSC security certification process in his "The Lesson of the Morris Worm" section, and I think he draws the wrong lessons from it by focusing on an incomplete certification and ignoring a completed one, as well as more recent academic research (langsec) that is becoming highly relevant in the age of LLMs. But first, one of his lessons (p. 43) is that his first question was malformed, that the Internet has to be open and purely a transport layer (the "end-to-end principle") and that the right question instead is to ask why endpoint computers are so insecure. That's clearly too quick a move, as we've seen countries build national firewalls and organizations deploy all manner of network-based security controls, and in the case of DDoS attack mitigation, they need to be outside of the targeted victim's network (preferably geographically distributed with AnyCast to diffuse the attack traffic and to filter it before it reaches the victim's network). We also should be free to raise questions about provider liability for being sources of attack (or even "attractive nuisances"), not blocked by an end-to-end principle that isn't actually adhered to in reality.
About the NCSC security certification process, Shapiro writes (pp. 44-45):
The story of the VAX VMM Security Kernel demonstrates the pitfalls of this strategy [assessing security and certifying]. In 1979, Major Roger Schell led a team to create an operating system that could withstand the NSA's most rigorous tests and achieve the highest possible score from the NSA--an A1 rating. To do so, his team built the system in a secured laboratory so that only the development group could enter. The machine they coded on--the development machine--was housed in a separate locked room within the lab. That locked room was protected by a cage. ...
It took a decade to build the system. By late 1989, the VMM Security Kernel was put in the field to undergo testing at government and aerospace installations. But in March 1990, DEC, the maker of the VAX minicomputer, canceled the project and removed prototypes from the testing sites. ...
He goes on to contrast with free and open-source software, noting the risk of backdoors and "Linus's law, which holds that with enough eyeballs, all bugs are shallow (i.e., easy to find)." We know that this is not quite accurate -- you need enough appropriately experienced eyeballs, pointed in the right places, which LLMs now have the potential to do, and seem to be ramping up rapidly on. But to my mind, he should have picked a different example than the VAX VMM Security Kernel, namely Honeywell's SCOMP (Secure Communications Processor) and its STOP operating system, which received an NCSC A1 security rating in 1985. When Groupe Bull acquired Honeywell's Large Computer Products Division in 1987-1991, the SCOMP division, Honeywell's Secure Computing Technology Center based in Minneapolis, was spun out into Secure Computing, and later acquired by McAfee. The STOP operating system was ported to other hardware and used for several military-grade communications gateways (and lessons learned from it were used in other products such as the Sidewinder firewall).
The main principle for an A1 security rating is that the system have "verified design," which includes formal specification, formal verification, mandatory and discretionary access control, configuration management, and distribution control. These are high-effort, resource intensive things to produce, but they are also things that are becoming more feasible to produce with LLM assistance (and perhaps necessary to produce to mitigate more vulnerabilities becoming shallow and discoverable, and possibly helpful in addressing limitations of LLMs themselves in a virtuous circle). It's worth noting here that in the Epilogue of the book, Shapiro gives an argument for the impossibility of "a magic app that checks programs for vulnerabilities" (p. 325), basically pointing out the issue of the halting problem and Turing computability. It's certainly correct that you can't build a system that will find all bugs, but it doesn't follow from that that you cannot build systems that find many bugs, nor that you cannot eliminate entire classes of vulnerability, nor that you cannot build specialized systems that have verifiable security properties. One of the lessons of langsec is that we can do things without Turing-complete systems, and we should do so, and this is something else that LLMs might make more feasible rather than relying on general purpose operating systems for every task and function. (Book recommendation for those who draw too-broad conclusions from theorems of undecidability, computability, and incompleteness: Torkel Franzen's 2005 book, Gödel's Theorem: An Incomplete Guide to Its Use and Abuse.)
To end on a positive note, I thought that Shapiro's coverage of the book's title hack, the Russian GRU ("Fancy Bear")'s breach of the DNC and DCC, was excellent in clearly laying out the details and timeline, and citing relevant sources that support the attribution. This occurs in chapters 7 and 8 (pp. 186-237), which should be mandatory reading for anyone who is still arguing that there is no evidence that Russia was behind the DNC hack or the subsequent data leaks in the face of massive evidence to the contrary. (It prompted me to add a reference to the book in my unpublished letter to the editor of Fortean Times in response to a columnist who should be mandated to read these chapters.)
Despite my issues with Shapiro's coverage of Multics and some of his conclusions, I enjoyed the book and recommend it for those interested in the history and context of the five hacks he covers.
Posts
