25 years of OpenBSD Security Tools: syslock and sysunlock

If you missed the overview post, you can see it here. This one is about managing immutable and append-only files on *BSD, Linux, and macOS.

Immutable and Append-Only Files

BSD-derived operating systems (including macOS) and Linux both support the concept of files being made immutable, so that neither their contents nor attributes can be changed. They also both support files being made append-only, so that the existing contents cannot be changed except by adding more data to the end. They do it in slightly different ways.

BSD Implementation

On BSD-derived systems, these features are controlled using file system flags which have "system" and "user" variants; the former can only be set and unset by the root user, while the latter can be set and unset by any user on files they own. The flags are changed using the chflags command, and their names are schg (system immutable), uchg (user immutable), sappnd (system append-only), and uappnd (user append-only). BSD-derived systems also take the further step for the system-level flags that they can be set but not unset after the system has booted from single-user mode to multi-user mode. This is controlled by the system security level (kern.securelevel) which is raised automatically by the init process from 0 ("Insecure Mode," also known as single-user mode) at initial boot to 1 ("Secure Mode," the default multi-user mode).  The system security level may also be raised with the sysctl command but cannot be lowered except by shutting the system down to return to single-user mode (securelevel=0), which shuts down system daemons and drops network connections, leaving the system accessible only by the system console until it reboots. The effect of this is that even the root user cannot modify the contents or attributes of files with schg (and can only append to files with sappnd) without shutting down the system.

Linux Implementation

On Linux, by contrast, only the root user can set or unset immutable or append-only attributes, using the chattr command with +i/-i or +a/-a to set or unset immutable or append-only attributes, respectively. But there is no system security level that prevents unsetting these attributes at any time after they've been set. The Linux attributes are equivalent to the BSD user flags, but limited to the root user. I have used syslock less on Linux than on OpenBSD, but have used it fairly broadly on Proxmox and Kali Linux.

macOS Implementation

On macOS, the BSD flags are present but by default the system is always in "Insecure Mode" (securelevel=0), and Apple has added additional file flags, notably restricted, which it uses on operating system commands and libraries as part of its "System Integrity Protection" (SIP) feature added in OS X El Capitan (10.11) in 2015. The restricted flag cannot be unset even when the system securelevel=0, but only when the system is booted into Recovery mode. While the restricted flag is, like the immutable and append-only flags, managed by the chflags command, it is not supported by my tools and it's just mentioned here to note that macOS has implemented a very similar capability in a different way. I have the least experience in using syslock on macOS, and because of that and the fact that system binaries are protected by this alternative mechanism, I use it less broadly there.

BSD Security Levels

When I first learned about file system immutable flags, the default system startup file on OpenBSD named /etc/rc.securelevel contained lines to set the system securelevel to 1, but that now happens automatically in the boot sequence by the init process, and the default rc.securelevel example file in /etc/examples consists only of comments and is no longer installed by default. All of my systems have that config file installed to set the securelevel, for reasons which will be explained shortly.

OpenBSD supports two other settings for kern.securelevel besides 0 and 1, which are -1 ("Permanently Insecure Mode") and 2 ("Highly Secure Mode"). Permanently Insecure Mode prevents the securelevel from being raised to 1 automatically; it instead goes to 0 and remains there.  Highly Secure Mode features all of the restrictions of Secure Mode, plus also restricts changes to host firewalls with OpenBSD's pf packet filter, allowing only changes to what IP addresses are in tables but no changes to filtering or NAT rules. The details of each level are documented in the OpenBSD securelevel man page. These restrictions are intended to reduce the impact and blast radius of both system compromises where an attacker gains root and to reduce the impact of administration errors.

Security Control, Administrative Safeguard, or Security Theater

The latter goal -- reducing the impact of administration errors -- is one that the BSD, Linux, and macOS default settings all support, but the former goal -- reducing the impact and blast radius of system compromises where an attacker gains root privileges -- is only evident for the BSD system immutable and append-only flags, where even the root user cannot unset them so long as the system is in Secure or Highly Secure mode. Some have argued that these flags are also mere speed bumps or error prevention (or "security theater"), on the grounds that they are easily bypassed, which can be done by returning the system to Insecure Mode. There are two main ways for a user with root access to do that, which are (1) using console access to access a root shell in single-user mode after shutdown, which may not be particularly easy as a remotely connected attacker likely does not have console access, or (2) modifying configuration files that allow command execution before the system raises the system securelevel, and rebooting the system. There are many configuration files and commands executed on the system during the boot sequence while the system is in Insecure Mode, and if any of those can be modified to either execute commands or prevent the system from going into Secure Mode, there is a path to unsetting the system immutable and append-only flags for the attacker (at the cost of a potentially noisy reboot). (I have made use of this path myself in the past while testing immutable flags and getting myself stuck.) It's because many files would have to be set immutable to close off that second reboot path that some have called even the system flags "security theater," but I think it is both feasible and it can be a genuine security control. The key is making it practical to lock enough of the right files, which is the problem I've tried to solve with these tools.

Origins of syslock/sysunlock

My syslock/sysunlock tool (two opposite functions in a single perl script, from here on I'll generally just refer to syslock except when the distinction matters) is designed to make the management of all of these implementations of immutable and append-only file system controls feasible, straightforward, and usable, at least as an error prevention method and at best as a security control. I originally wrote it after coming across a simple shell script of the same name by George Shaffer, but it now looks quite different, supporting system and user immutable and append-only flags on OpenBSD and macOS, and Linux's near equivalent (+i/-i and +a/-a). My recommendation for anyone starting out on a BSD system is to begin with the user flags, which are trivial to unlock and cause no permanent damage if you need to make changes quickly. Once comfortable with the group structure, you can consider adding system flags for files that rarely need to change. I'll describe that group structure next.

syslock Groups

The main feature of syslock that makes it usable is that it is configured to place lists of files and directories into groups, and those groups can be locked or unlocked with a single command (syslock -g <groupname>). The groups are defined like tags associated with a list of files and directories, so that a given file or directory can be in multiple groups. The default configs supplied with the tool include group names such as etc (files in /etc), etcrare (files in /etc that are rarely modified), and fstab (/etc/fstab gets its own group as it's a painful file to clobber by accident). There are relatively self-explanatory group names like binaries, libraries, and system, and then there's presecure, which covers the files and directories that are potential targets for someone trying to find a way to bypass system immutable flags (e.g., startup scripts like /etc/rc, /etc/rc.local, /etc/rc.d, and others that may be less obvious like /etc/sysctl.conf).  (There is no presecure group defined in the Linux or macOS example configs.) For BSD and macOS, you can choose to make user immutable or system immutable your default, but you can also use groups to explicitly call out a set of files as the opposite of your default, with the group names schg and uchg (and similarly with sappnd and uappnd groups). These groups are specially handled so that they can be specified in combination with another group name (e.g., etc:uchg or acct-logs:uappnd) to identify the files and directories that are in both.

Since it is possible to enable the schg flag in Secure Mode but not to disable it, by default neither syslock nor sysunlock will touch those files while in Secure Mode (sysunlock can't unlock them), but the -f (force) option to syslock will lock them. A group can have both a system and a user subset of files regardless of what default is configured. For example, if your default is schg but you want some commonly modified files in /etc to be uchg, you could put them in that group along with the etc  group. Then, if you unlocked files in Insecure Mode with -g etc, both the schg and uchg files in the etc group would be unlocked, while if you used sysunlock -g etc in Secure Mode, only the uchg files would be unlocked. If you were in Insecure Mode but only wanted to lock the uchg files in the etc group, you can use -g etc:uchg.

The intention of groups is to provide a mechanism for unlocking specifically what is needed to perform a specific task. The default and sample config files supplied with the tool include groups for other tools that will be covered in this series of blog posts, including reportnew (blog post June 16), rsync (for rsync-tools, blog post June 9), and sigtree (blog post June 19), as well as for process accounting log files (to make them append-only), mail servers, web servers, and DNS servers (the latter two in the BSD default config only). Log files and DNS servers provide two examples for where you'd want to use different flag-specific subgroups. For log files, you want the active log file to be append-only, but you want the rotated and archived log files to be immutable; you also want them to use user rather than system flags if the logs are subject to rotation via newsyslog or other log rotation mechanism. Thus the sample configs put the live process accounting log in the uappnd group and the rotated process accounting logs in the uchg group. The rotation process needs to unlock both of those subgroups before rotation, and lock them again after rotation. For DNS configuration files, you might want to lock your zone files with schg if they don't change frequently and aren't changed by any automation, but if you use automated DNSSEC signing, you'd lock the signed zone files with uchg so that they can be unlocked before signing and relocked after signing. It's worth noting that logging to a separate machine is a much better security control than using uappnd (or Linux +a) flags on log files--but I do both.

Overlapping Groups

Note that it is possible to define groups that overlap with each other in various ways, and while this is generally acceptable and unlikely to cause any issues on Linux or if only a single type of flag is used on BSD or macOS, it can create issues if overlapping groups have a mix of flag types such that multiple flags get set on any files or directories. The default and sample configs do not contain any group definitions that create this problem, and syslock will generate warnings if it detects cases of conflicting flag types in the configuration, but will not detect all possible cases (e.g., where symlinks are involved).

OpenBSD-specific: KARL Features

There is also an implicit group of files identified in the config file by a leading "!" prefix character, which designates system files that are part of OpenBSD's Kernel Address Randomized Link (KARL) feature, where the kernel and key binaries (currently libc, libcrypto, ld.so, sshd, sshd-session, sshd-auth, and ssh-agent -- the list may increase between releases) are relinked in random order at reboot.  These files are locked or unlocked by using a -s (for system) option. (This was perhaps a poor choice of option letter and name, as it is distinct from the "system" group which is intended to capture key operating system files and what needs to be unlocked for upgrading and patching, though I typically unlock everything before upgrading or patching. This is just to ensure that nothing the system needs to install is blocked, which can leave a system in an inconsistent and not fully operable state, but the "system" group should actually work for this purpose.) Since I use system immutable flags as my default on most systems, and on the presecure group on all systems, my practice for patches and upgrades is to shut the system down (enter Insecure Mode), unlock everything (sysunlock with no arguments), perform the upgrade (with an extra step) or patch, then lock everything (syslock with no arguments), then unlock what's needed for Karl (sysunlock -s), and then exit, which starts up system daemons and networking, and returns to Secure Mode, without rebooting the kernel or resetting the system uptime counter. In my rc.securelevel file I have the following line:

echo -n ' running syslock'; (/bin/sleep 10; /usr/local/bin/syslock -swf) &

The -s and -f options have already been explained, but -w means to wait for KARL relinking to complete before locking, so as not to prevent that process from occurring. Typically, I do not unlock with -s for most instances when I shut down to make changes, and so I do prevent KARL relinking in those cases, and it doesn't lead to any system inconsistencies--it just preserves the previously established link ordering.

The extra step I mentioned above for an upgrade is that I comment out the line in rc.securelevel that re-locks the system because at the completion of a sysupgrade process there will be a reboot after which I will want to make many additional file changes using sysmerge to update system configuration files, update packages I have installed, remove unused binaries and old libraries, and so forth.  I then remove the comment and reboot again. (Also, if a patch with syspatch rebuilds the kernel, I'll reboot with the new kernel after the return to Secure Mode and the KARL process completes.)

One side note is a little trick I use on my OpenBSD firewall to avoid unnecessary disruption to traffic when doing patching and software updates that require unlocking system immutable flags. The trick is that shutting down to Insecure Mode doesn't impact traffic routing or packet filtering, but running netstart when you exit back to Secure Mode does. (Disruption is unavoidable if it's a kernel patch that requires a full reboot.) The trick is that I've modified my /etc/rc (something I generally avoid doing for ease of upgrades, so I keep the original) by adding one line before and two lines after the call to netstart:

if [ ! -f /var/run/no_netstart_resume ]; then

    sh /etc/netstart

    echo "bypassing netstart"

    /bin/rm -f /var/run/no_netstart_resume

With this in place, I touch /var/run/no_netstart_resume before I exit, and the system restarts with no network disruption.

Audit Feature

Both syslock and sysunlock have an audit (-a) option, which will report which files are currently not in the expected state. That is, syslock -a will tell you all the files which should be locked per the config but are unlocked, and sysunlock -a will tell you all the files which should be unlocked per the config but are locked. This can also be applied to any specific group with -g <groupname>. If used on a BSD system in Secure Mode, the audit can be restricted to what would actually be changed at the current security level using the -o (operational restrictions) option. A -q (quiet) option will suppress all output and just return 0 for success or 1 (error) for failure; this was created to allow a check to see if all files in a particular group are unlocked for an installation to occur for my install.pl tool (to be covered in a June 12 blog post). With -q, the audit will finish and return at the first discrepancy found.

Path Prefixes

The configuration file syntax allows listed paths to use three other prefixes in addition to the OpenBSD-specific "!" referenced above. These prefixes, which only have effect on directory paths (and generate a warning but are otherwise ignored on other file types) are:

+  Do not recurse through subdirectories.  (No prefix, the default, means lock the directory and every thing in it, recursively.)

-  Do not lock the directory itself, just its contents.

=  Lock the directory and its file contents, but not subdirectories.

These cannot be used in combination (though some combinations, like =-, might conceivably be useful), and while I initially made use of the first two path prefixes in my configs, they ended up not being particularly useful with the patterns of groups I developed. I recently added the = path prefix to address a Linux-specific case.

Linux-specific: Bootloader Considerations

On many Linux systems (notably on Proxmox which uses Debian), the grub bootloader rewrites files in /boot/grub, (specifically grub.cfg and grubenv) on reboots. If those files are immutable, the reboot will fail to update those files. The default Linux config creates a grub group for /boot/grub and its contents, and uses the = path prefix on a separate group for /boot and its other subdirectories. The grub group can then be unlocked before a reboot, while the broader group is unlocked for kernel updates, and both locked again afterward. The grub group unlocks slightly more than strictly necessary (everything in /boot/grub rather than the directory and the two specific files), but handles the common case cleanly.

Use of Large Language Models (LLMs)

I've made use of LLMs, initially for performing security assessments with suggested improvements that I'd implement selectively and by hand, but subsequently for working out the details of design for prospective changes, writing code, and identifying the causes of bugs. Each of these capabilities has significantly improved since the initial use for code assessments in the summer of 2025. Specific capabilities implemented by Claude include adding append-only flag support, improving error messages and warnings, code refactoring, enhancing the audit features and adding the -o option, and drafting the Github README. Most recently, while I was in the process of writing this blog post, I revisited an issue I had run into involving overlapping groups with differing flags, and thought that a new path prefix that ignored directories might be a good way to handle it. In addition to that specific use case, for which Claude suggested the use of = (to mean "lock the directory and files at the same level"), I asked whether any combinations of existing path prefixes might be useful or if any other new prefixes might make sense. Claude suggested that -= or =- might be meaningful and useful (don't lock the directory itself, lock the files in it but not the subdirectories), but I chose not to add the additional complexity without a specific use case. In the process I identified a bug in how path prefixes were being handled in previous Claude-generated code related to perl taint handling, and Claude fixed it along with the implementation of =.

I've used ChatGPT and Gemini in addition to Claude for security assessments on the code, and they've each identified different issues. While each has found real bugs, my impression is that I've seen more false positives from ChatGPT and Gemini than from Claude, though they've also found real bugs in code written by Claude.

I've found LLMs quite useful for security auditing, refactoring, and error message improvements, as well as writing new capabilities that are relatively straightforward. In the case of implementing append-only flags, it took multiple design, build, and test cycles to get the code to production quality. Overall, forcing myself to explain design choices clearly (and sometimes reconsider them) in the process of using LLMs has been extremely valuable, and has also been a benefit of writing this blog post. I used Claude to identify key topics that should be mentioned in this series, but did not rely on Claude to outline or write the post. Claude and a human editor reviewed this post (and the opening overview post) identify typos and suggest edits.

I'll include a section like this in each post in the series.

Getting Started and Further Reading

The sample config files supplied with syslock cover OpenBSD, Linux (with some Proxmox specifics in comments), and macOS, and provide a reasonable starting point. As noted above, I recommend starting with uchg groups on BSD and macOS, which gives you immediate error prevention value and a chance to discover what needs to be unlocked for your specific regular workflows before committing to more restrictive schg flags. For Linux, there's only one kind of immutable attribute and it's changeable without a shutdown, so you can start immediately with the sample config.

As noted above, the syslock audit feature is integrated with my install.pl tool and syslock groups more generally are a key feature of both that tool and its distribute.pl counterpart, both of which will be covered in a blog post on June 12. The wrap-up post on June 23 will show how all of these tools fit together into a coherent security architecture.

syslock is available on my website and on Github, along with sample configs for OpenBSD, Linux, and macOS.

25 years of OpenBSD security tools

I've been using and administering OpenBSD systems since 1999 (OpenBSD 2.5). During that time, I've written numerous scripts to make things easier, more automated, or more secure, or sometimes just to improve my understanding of how things work. When I started managing my home systems, I ran several Internet-exposed services on my home network (DNS, mail, web, SSH). I used djbdns, qmail, and Apache httpd at the start before switching to nsd/unbound for DNS and postfix for mail, and finally to OpenSMTPD for mail. When I got tired of excessive inbound traffic I moved my authoritative DNS to a provider while keeping an internal zone and resolvers, set up two cloud servers for mail and my public webserver. My home network became a hardened, minimal-exposure architecture that only allows Wireguard from expected sources and mail (after mutual TLS authentication with certificates) while continuing to run internal services.

Over the years I've made a number of these scripts available via my website and on Github once they were sufficiently mature, configurable, and with reasonable defaults in supplied sample configuration files. A set of them fit together in a coherent framework that I thought I'd write about in a series of blog posts for those who might be interested in either using them or learning from them. All are under BSD licenses which require no permission from me to use or rework into something else -- my intent is not to obtain a new user base that I need to support, though I'm happy to consider suggested enhancements.

The tools I will cover in these posts, while written for OpenBSD originally, also work on Linux and macOS, and likely on other BSDs. I use most on Kali and Proxmox (Debian Linux), as well as a few on macOS (which has roots in FreeBSD). All are written in perl and use OpenBSD-specific features when available (such as pledge and unveil, covered in an earlier blog post). Several use privilege separation to run as much functionality as possible as a non-privileged user.

The earliest tools, written initially between 1999 and 2004, were intended to provide log monitoring, file integrity monitoring, and to manage file system flags for making files immutable, a feature I hadn't seen widely used until macOS introduced a variant in 2015. Even among OpenBSD users, immutable flags are not commonly used and I recall early warnings that they were difficult to use because many files would have to be made immutable in order to avoid somewhat trivial bypasses.

The later-developed tools in the set I will cover, written in 2022-2024, are for file and software distribution across hosts and providing a perl interface to OpenBSD's signify cryptographic signing and verification tool for use by most of the tools in the set.

I'll note in my discussion how I've used large language models (LLMs) for security assessment, bug fixing, and enhancement on these tools, and where it has been useful and some of the obstacles I've run into along the way.  I'll also describe how they work together in various ways, with a comprehensive view in the final wrap-up. Here's the planned schedule:

* June 2: this overview post

* June 5: syslock and sysunlock (initially written 2004, inspired by a much simpler tool called syslock by George Shaffer).

These tools allow you to define and manage groups of files which use BSD system and user immutable and append-only flags or Linux immutable or append-only flags. The use cases range from genuine security enhancement to error prevention.

* June 9: rsync-tools (rsync-client.pl/rsync-server.pl, initially written 2003; rrsync, a 2022 fork of the version initially written by Joe Smith in 2004 and modified by Wayne Davison, the primary developer of rsync; rsync-altroot.pl initially written 2002).

This is a collection of tools used with rsync to define sets of files to be kept synchronized between hosts, perform backups, and to place restrictions on rsync using various mechanisms.

* June 12: distribute.pl/install.pl (originally written 2022) and some extras (including Signify.pm, originally written 2024)

Two tools for distributing content (plain files, config files, certificates, and signed OpenBSD-style packages) to multiple hosts, verifying signatures and installing on the remote hosts. These are not limited to use on OpenBSD, as the install tool will work on Linux and macOS as well--so long as the content being distributed is usable on those operating systems, which includes architecture/OS-independent OpenBSD-style packages such as perl scripts--like all of the scripts in this blog series, which are available as signed OpenBSD-style packages. These are packaged with some extras (gendoas.pl with distribute.pl and pkg_info.pl with install.pl) and this blog post will also cover Signify.pm, a perl interface to OpenBSD's signify.

* June 16: reportnew (originally written 1999)

A log monitoring tool, inspired by swatch but written to support djbtools' cyclog and multilog format logs, Linux journal files, as well as BSD, Linux, and macOS process accounting logs, the latter of which may be a feature unique to this tool (I've not seen it in any other log monitoring tool).

* June 19: sigtree.pl (originally written 2000)

A file integrity monitoring tool, inspired by the original tripwire.

* June 23: Wrap-up post

Each post will discuss the origins of the tool(s), the use cases and problems solved, the security model, the limitations, and, in some cases, what might be added in the future.  As each post is published, links will be added above.

If you want to skip ahead and look into these tools further, all are available on Github and on my website.

Book Review: Scott J. Shapiro, Fancy Bear Goes Phishing: The Dark History of the Information Age, in Five Extraordinary Hacks

 Scott Shapiro's 2023 book's aim is to answer three questions: (1) why is the Internet (still) so insecure? (2) how do hackers do what they do? and (3) what can be done about it? He recounts some historical events, the "five extraordinary hacks" of the subtitle, to tell the story, and introduces the terms "upcode" and "downcode" as the core concepts in his framework for understanding--where "downcode" means actual, implemented computer code and "upcode" means the social, political, and institutional forces providing incentives and governance.  This is essentially a simplified version of Lawrence Lessig's four forces of law, social norms, markets, and code spelled out in his 1999 book, Code: And Other Laws of Cyberspace, and also reminded me of the framework in Bruce Schneier's 2012 book, Liars and Outliers: Enabling the Trust That Society Needs to Thrive, where the four forces are moral pressures (internalized incentives), social pressures (social/cultural incentives from other people), institutional guidelines and rules (formal rules, regulations, and laws), and security systems (locks, police, firewalls, fraud detection, etc. -- actual operational controls which may be implemented physically, in code, or by policies and practices). For Shapiro, Lessig's first three forces are "upcode" and only code is "downcode," and Schneier's first three forces and parts of his fourth are "upcode."

I found Shapiro's answers to the first and last questions of his project fairly unsatisfying by comparison to Schneier's book (or to, say, Ross Anderson's massive Security Engineering book, which you can find free online and which is the number one book I recommend to people interested in computer security). His upcode/downcode framework is, I think, too simplistic, and while he correctly identifies some of the specific factors relevant to his five extraordinary hacks, he also makes some bad arguments and gets some things quite wrong, and I think misses some of the key issues that have become highly relevant today with the rise of large language models (LLMs).

His answers to the second question, specific to the hacks he describes, however, are generally quite good and detailed, and this is where I think the book is at its strongest.  The five hacks are:

(1) The Robert Morris Jr. worm.  (2) Dark Avenger's viruses.  (3) The breach of Paris Hilton's T-Mobile Sidekick. (4) The Russian GRU (CrowdStrike name: "Fancy Bear") breach of the DNC and DCCC.  (5) The DDoS attacks of the Mirai botnet.

Each hack is presented over multiple chapters, with historical context and explanation of technical details, as well as relevant social and psychological (and legal) factors, in his upcode/downcode framework. I share a complaint with other reviews that Shapiro has a tendency to invent unnecessary terminology ("vorms" for a virus/worm hybrid, and "mudges" for malicious nudges, of the sort commonly referred to as "dark patterns" or that are examined in depth in Natasha Schüll's 2012 book, Addiction by Design: Machine Gambling in Las Vegas).

My biggest factual complaints with Shapiro come in some of the background context he supplies to the 1988 Morris worm, when he gives a brief history of the UNIX operating system that begins with its predecessor, Multics. Shapiro notes that his father, like Morris's, worked at Bell Labs. (And disclosure: I used Multics beginning around 1978-79 and as a Multics developer from 1983-1989.) Bell Labs is the likely source of the issues in Shapiro's discussion of Multics. Shapiro describes the history of Multics like this (p. 26):

A large team from MIT, Bell Labs, and General Electric, therefore, decided to develop a complete multiuser operating system as a replacement for batch processing. They called it Multics, for Multiplexed Information and Computing Service.

   The Multics team designed its time-sharing with security in mind. Multics pioneered many security controls still in use today--one of which was storing passwords in garbled form so that users couldn't repeat Allan Scherr's simple trick. After six years of development, Multics was released in 1969.

   The military saw potential in Multics. Instead of buying separate computers to handle unclassified, classified, secret, and top-secret information, the Pentagon could buy one and configure the operating system so that users could only access information for which they had clearance. The military estimated that it would save $100 million by switching to time-sharing.

   Before the air force purchased Multics, they tested it. The test was a disaster. It took thirty minutes to figure out how to hack into Multics, and another two hours to write a program to do it. "A malicious user can penetrate the system at will with relatively minimal effort," the evaluation concluded.

   The research community did not love Multics either. Less concerned with its bad security, computer scientists were unhappy with its design. Multics was complicated and bloated--a typical result of decision by committee. In 1969, part of the Multics group broke away and started over. This new team, led by Dennis Ritchie and Ken Thompson, operated out of an attic at Bell Labs using a spare PDP-7, a "minicomputer" built by the Digital Equipment Corporation (DEC) that cost ten times less than an IBM mainframe.

   The Bell Labs team had learned the lesson of Multics' failure: Keep it simple, stupid.

Based on this description, one might infer that the US Air Force did not purchase Multics and that it faded away, but in fact they purchased multiple Multics systems and operated them for decades. One might also infer that the Paul Karger and Roger Schell-led penetration test was the end of the story, when it was just the beginning--the issues identified were corrected, and on September 1, 1985, Multics was the first operating system to receive a B2 security rating from the National Computer Security Center (a project I participated in). Shapiro's "thirty minutes to figure out how to hack" and "another two hours to write a program to do it" is referring to the "Master Mode Transfer" vulnerability in table 3 of the Karger and Schell report, and it should be noted that these time estimates are for a computer science graduate of a major university with moderate knowledge of Multics design and nine months experience as a Multics programmer with access to the source code (which Multics provided to its customers for all except the "unbundled" extra-cost software). Shapiro's "A malicious user can penetrate the system at will with relatively minimal effort" is an accurate quote from the report, but not from its conclusion, rather, from an earlier section titled "Multics as a Base for a Secure System" that is referencing the prior section, "Multics is not Now Secure." The next section says that "In the long term, it is felt that Multics can be developed into an open secure multi-level system by restructuring the operating system to include a security kernel." And that was, in fact, done -- the B2 security rating was earned, in part, for demonstration of the security of the Multics Trusted Computing Base (TCB), aka, the "kernel."

Shapiro's description says that Multics was "complicated and bloated." While it is true that some joked that Multics stood for "many unnecessary large tables in core [memory] simultaneously," it was incredibly small and streamlined by modern standards. Karger and Schell's followup paper, "Thirty Years Later: Lessons from the Multics Security Evaluation," notes that the 1973 Multics TCB was 628K bytes, and that the security policy alone of SELinux is 2.5 times larger, not even counting the Linux kernel itself.

It is also worth noting that while Shapiro's description suggests that the Bell Labs team broke away and started afresh after this failure, Bell Labs actually withdrew from Project MAC very early in the project, in March 1969, while the Air Force assessment took place from July 1972 to October 1973. Ken Thompson has admitted in an interview that he's never used a Multics system.

It is undeniably true that UNIX outcompeted Multics, but the reasons for this are many and include that Honeywell had three mainframe operating systems, GCOS (formerly GECOS), Multics, and CP-V (and its successor CP-6), and GCOS was the one that was given the most support, development, and marketing resources. Unlike UNIX, which became open source in the early nineties and its essentials cloned as the open source Linux, Multics was not made open source until November 2007 and never ported to alternative hardware. I highly recommend that anyone interested in operating system history and computer security spend some time perusing Tom Van Vleck's Multicians website, as there are still lessons to be learned from and about Multics. (And you can now run one yourself on a laptop or even a Raspberry Pi, via Honeywell DPS8 hardware emulation.)

A few pages later (pp. 29-30), Shapiro writes again about Multics, and rightly gives credit to the Karger and Schell report for something UNIX co-creator Ken Thompson regularly is given credit for coming up with:

When Ken Thompson won the Turing Lifetime Achievement Award in 1984, the highest honor in the computer-science community, for developing UNIX, he devoted his lecture to cybersecurity, a first for the Turing lecture. In the first half of his lecture, Thompson describes a devious hack first used by air force testers when they penetrated the Multics system in 1974. They showed how to insert an undetectable "backdoor" in Multics. ... The moral Thompson drew was bracing: the "only program you can truly trust is the one you wrote yourself."

Thompson's lecture, published as "Reflections on Trusting Trust," ends with an acknowledgement in footnote four that he didn't come up with this idea, citing "Unknown Air Force Document" (which is the first Karger and Schell document linked above). The specific idea is not just a trapdoor or trojan horse, but a trapdoor in the compiled code that is not present in the source code, but is inserted by the compiler itself -- including that insertion functionality itself being automatically inserted into the compiler when the compiler itself is recompiled. The idea is that the trapdoor cannot be observed in the source code, it's not present there, but is only in the compiled binary. Karger and Schell describe it like this (in section 3.4.5, "Trap Door Insertion"):

Clearly when a trap door is inserted, it must be well hidden to avoid detection by system maintenance personnel. Trap doors can best be hidden in changes to the binary code of a compiled routine. Such a change is completely invisible on system listings and can be detected only by comparing bit by bit the object code and the compiler listing. However, object code trap doors are vulnerable to recompilations of the module in question. ...

It was noted above that while object code trap doors are invisible, they are vulnerable to recompilations. The compiler (or assembler) trap door is inserted to permit object code trap doors to survive even a complete recompilation of the entire system In Multics, most of the ring 0 supervisor is written in PL/I. A penetrator could insert a trap door in the PL/I compiler to note when it is compiling a ring 0 module. Then the compiler would insert an object code trap door in the ring 0 module without listing the code in the listing. Since the PL/I compiler is itself written in PL/I, the trap door can maintain itself, even when the compiler is recompiled. 

An object code compiler trap door, therefore, would mean that you can't even trust a program you wrote yourself! There is, however, a route to addressing this problem, which was the subject of David A. Wheeler's doctoral dissertation in 2009, "Fully Countering Trusting Trust Through Diverse Double-Compiling." (Note that Karger and Schell did demonstrate an object code trap door in the Multics penetration test, but not a compiler object code trap door.)

Shapiro does mention the NCSC security certification process in his "The Lesson of the Morris Worm" section, and I think he draws the wrong lessons from it by focusing on an incomplete certification and ignoring a completed one, as well as more recent academic research (langsec) that is becoming highly relevant in the age of LLMs.  But first, one of his lessons (p. 43) is that his first question was malformed, that the Internet has to be open and purely a transport layer (the "end-to-end principle") and that the right question instead is to ask why endpoint computers are so insecure. That's clearly too quick a move, as we've seen countries build national firewalls and organizations deploy all manner of network-based security controls, and in the case of DDoS attack mitigation, they need to be outside of the targeted victim's network (preferably geographically distributed with AnyCast to diffuse the attack traffic and to filter it before it reaches the victim's network). We also should be free to raise questions about provider liability for being sources of attack (or even "attractive nuisances"), not blocked by an end-to-end principle that isn't actually adhered to in reality.

About the NCSC security certification process, Shapiro writes (pp. 44-45):

The story of the VAX VMM Security Kernel demonstrates the pitfalls of this strategy [assessing security and certifying]. In 1979, Major Roger Schell led a team to create an operating system that could withstand the NSA's most rigorous tests and achieve the highest possible score from the NSA--an A1 rating. To do so, his team built the system in a secured laboratory so that only the development group could enter. The machine they coded on--the development machine--was housed in a separate locked room within the lab. That locked room was protected by a cage. ...

    It took a decade to build the system. By late 1989, the VMM Security Kernel was put in the field to undergo testing at government and aerospace installations. But in March 1990, DEC, the maker of the VAX minicomputer, canceled the project and removed prototypes from the testing sites. ...

 He goes on to contrast with free and open-source software, noting the risk of backdoors and "Linus's law, which holds that with enough eyeballs, all bugs are shallow (i.e., easy to find)." We know that this is not quite accurate -- you need enough appropriately experienced eyeballs, pointed in the right places, which LLMs now have the potential to do, and seem to be ramping up rapidly on. But to my mind, he should have picked a different example than the VAX VMM Security Kernel, namely Honeywell's SCOMP (Secure Communications Processor) and its STOP operating system, which received an NCSC A1 security rating in 1985. When Groupe Bull acquired Honeywell's Large Computer Products Division in 1987-1991, the SCOMP division, Honeywell's Secure Computing Technology Center based in Minneapolis, was spun out into Secure Computing, and later acquired by McAfee. The STOP operating system was ported to other hardware and used for several military-grade communications gateways (and lessons learned from it were used in other products such as the Sidewinder firewall).

The main principle for an A1 security rating is that the system have "verified design," which includes formal specification, formal verification, mandatory and discretionary access control, configuration management, and distribution control. These are high-effort, resource intensive things to produce, but they are also things that are becoming more feasible to produce with LLM assistance (and perhaps necessary to produce to mitigate more vulnerabilities becoming shallow and discoverable, and possibly helpful in addressing limitations of LLMs themselves in a virtuous circle). It's worth noting here that in the Epilogue of the book, Shapiro gives an argument for the impossibility of "a magic app that checks programs for vulnerabilities" (p. 325), basically pointing out the issue of the halting problem and Turing computability. It's certainly correct that you can't build a system that will find all bugs, but it doesn't follow from that that you cannot build systems that find many bugs, nor that you cannot eliminate entire classes of vulnerability, nor that you cannot build specialized systems that have verifiable security properties. One of the lessons of langsec is that we can do things without Turing-complete systems, and we should do so, and this is something else that LLMs might make more feasible rather than relying on general purpose operating systems for every task and function. (Book recommendation for those who draw too-broad conclusions from theorems of undecidability, computability, and incompleteness: Torkel Franzen's 2005 book, Gödel's Theorem: An Incomplete Guide to Its Use and Abuse.)

To end on a positive note, I thought that Shapiro's coverage of the book's title hack, the Russian GRU ("Fancy Bear")'s breach of the DNC and DCC, was excellent in clearly laying out the details and timeline, and citing relevant sources that support the attribution. This occurs in chapters 7 and 8 (pp. 186-237), which should be mandatory reading for anyone who is still arguing that there is no evidence that Russia was behind the DNC hack or the subsequent data leaks in the face of massive evidence to the contrary. (It prompted me to add a reference to the book in my unpublished letter to the editor of Fortean Times in response to a columnist who should be mandated to read these chapters.)

Despite my issues with Shapiro's coverage of Multics and some of his conclusions, I enjoyed the book and recommend it for those interested in the history and context of the five hacks he covers.

Books read in 2025

   Not much blogging going on here still, but here's my annual list of books read for 2025.

• Adam Becker, More Everything Forever: AI Overlords, Space Empires, and Silicon Valley's Crusade to Control the Fate of Humanity
• Rutger Bregman, Humankind: A Hopeful History (2019)
• Samuel D. Brunson, Between the Temple and the Tax Collector: The Intersection of Mormonism and the State
• Kate Conger and Ryan Mac, Character Limit: How Elon Musk Destroyed Twitter (2024)
• Mark Jonathan Davis, Grateful: 25 Years of Music, Movies, and Medical Emergencies with Richard Cheese & Lounge Against the Machine, Part One: Stranger in a Strange Lounge
• Renée DiResta, Invisible Rulers: The People Who Turn Lies Into Reality (2024)
• Cory Doctorow, Picks and Shovels: A Martin Hench Novel
• Erle Stanley Gardner (Martin H. Greenberg and Charles G. Waugh, eds), The Human Zero: The Science Fiction Stories of Erle Stanley Gardner (1981)
• Brooke Harrington, Offshore: Stealth Wealth and the New Colonialism (2024)
• Gabriel Kennedy, Chapel Perilous: The Life & Thought Crimes of Robert Anton Wilson (2024)
• Thomas Levenson, So Very Small: How Humans Discovered the Microcosmos, Defeated Germs--and May Still Lose the War Against Infectious Disease
• Mary Roach, Replaceable You: Adventures in Human Anatomy
• Oliver Sacks, The Island of the Colorblind (1996)
• Oliver Sacks, The Mind's Eye (2010)
• Neil Sheehan, A Bright Shining Lie: John Paul Vann and America in Vietnam (1988, 2009 edition)
• Quinn Slobodian, Hayek's Bastards: Race, Gold, IQ, and the Capitalism of the Far Right
• Dana Stevens, Camera Man: Buster Keaton, the Dawn of Cinema, and the Invention of the Twentieth Century (2023)
• Katherine Stewart, Money, Lies, and God: Inside the Movement to Destroy American Democracy
• Spencer Sunshine, Neo-Nazi Terrorism and Countercultural Fascism: The Origins and Afterlife of James Mason's Siege (2024)
• Sam Tanenhaus, Buckley: The Life and the Revolution That Changed America
• Mark S. Weiner, The Rule of the Clan: What an Ancient Form of Social Organization Reveals About the Future of Individual Freedom (2013)
• Tim Weiner, The Mission: The CIA in the 21st Century
• Lawrence Wright, The Looming Tower: Al-Qaeda and the Road to 9/11 (2006)
• Sarah Wynn-Williams, Careless People: A Cautionary Tale of Power, Greed, and Lost Idealism

Top for 2025 published in 2025: Tanenhaus, Levenson, Roach, Weiner, Davis, Wynn-Williams, Becker, Doctorow; other top reads for the year: Sheehan, M. Weiner, Sacks

A few planned or already (or still) in-progress reads for 2026:

Robert Caro, The Power Broker: Robert Moses and the Fall of New York (1975)

G.A. Cohen, Self-Ownership, Freedom, and Equality (1995)

John Ferris, Behind the Enigma: The Authorised History of GCHQ, Britain's Secret Cyber-Intelligence Agency (2020)

Peter H. Wilson, The Holy Roman Empire: A Thousand Years of Europe's History (2017)

Arthur M. Melzer, Philosophy Between the Lines: The Lost History of Esoteric Writing (2014)

(Previously: 2024, 2023, 2022, 2021, 2020, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005.) 

Comment on Steve Novella's "Rethinking the Skeptical Movement" a decade ago

 I just came across this comment I wrote a decade ago on a post that Steve Novella wrote on his blog, and I think it's pretty good, but it generated zero comment and no upvotes or downvotes. I just came across it again while looking for old comments I made about Al Seckel, who is in the news again for his role in attempting to scrub negative information about Jeffrey Epstein from the Internet.

lippard10 years ago

This sentence contains an argument in which the conclusion does not follow from the premises:

"The Committee for Skeptical Inquiry (CSI) is one of the oldest standing skeptical organizations and they have editors, fellows, and advisory committee (of which I am a member) and therefore have the ability to maintain high levels of quality within their own sphere."

The history of organized skepticism shows that CSICOP has repeatedly run into issues of ethics, poor methodology, and fraud by its Fellows and associates that has only been dealt with because of external pressures. That includes the Mars effect controversy, plagiarism by Robert Baker in the pages of Skeptical Inquirer, credential misrepresentation by Al Seckel, and the Uri Geller lawsuits (which had nothing to do with whether or not he actually had psychic capabilities, though they are often misrepresented by skeptics as though that was what the lawsuits were about).

The skeptical movement arose in the U.S. in the 1970s as a counter-movement, as a response to an increase in belief in the occult, new religions, and the "New Age" movement. It has periodically resurged in response to various other challenges--faith healing televangelists, creationist legislation, parapsychology's finding du jour, and so on. But it seems to me that it has largely been reactionary and not a self-sustaining movement. Although it is supposedly a scientific movement, the choice was made at the beginning to address a popular audience, and, after the Mars effect, not to directly fund or sponsor scientific research. This was partly reversed as CSI "Research Fellows" were appointed, but most of their work tends to be historical or in response to specific popular claims, as opposed to experimental work--and it tends to be published in popular journals, not scientific ones.

CSI Fellows are not members of the organization and have no voting power or control over the organization, apart from those who are members of the Executive Council. They are, for the most part (with a few exceptions), famous figureheads who do not directly contribute research or work to the organization, but merely lend their reputations to the group for purposes of self-promotion. By contrast, the Parapsychological Association (for one example, the Society for Scientific Exploration is another) is an organization of practicing scientists, doing research in the area, who publish in an academic-style, peer-reviewed journal. The PA, unlike CSI, is a member of the AAAS.

CSICOP/CSI has published a series of goals and objectives in the Skeptical Inquirer over its history, and it has clearly achieved some of those goals (like the original "To establish a network of people interested in examining claims of the paranormal"--changed from "establish" to "maintain" in 1980--and "To convene conferences and meetings"), failed at others (like "To prepare bibliographies of published materials that carefully examine such claims", a goal deleted in 1998; and "To encourage and commission research by objective and impartial inquirers in areas where it is needed"--the "commisions" was removed in 1994). However, CSI removed most of these objectives from the Skeptical Inquirer in 2001, and hasn't listed any at all in the magazine since 2009.

A clear vision, mission, goals, and objectives are necessary for an effective organization.

Illinois state representative Mike Bost's dog-killing story

 Another case of conservative animal abuse, via libraryjayne on Threads, Illinois state representative Mike Bost (R-Murphysboro):

The earliest episode dates back to 1986, when a neighborhood beagle named Rusty bit Bost's 4-year-old daughter. The report filed by animal control officials indicates that the girl provoked the attack by chasing the dog. She ultimately had to get 19 stitches on her face.

According to court records, Bost was displeased that authorities would not be able to deal with the 10-year-old dog immediately. So he got his handgun, drove to Rusty's owner's home, and shot the dog to death while it was penned in an enclosure.

Neighbors were "very alarmed and disturbed," according to the police report, but a jury eventually found Bost not guilty of breaking any laws. The local paper reported the case under the headline "Area man acquitted in dog killing trial."

(From Michael McAuliffe at HuffPo, Sep 26, 2014.)

Causing unnecessary death and suffering

 If your reasons for voting for Donald Trump for president included that you wanted to cause unnecessary death and suffering and reduce to the standing and trustworthiness of the United States with the rest of the world then congratulations, you've been given what you wanted. If not, maybe you should engage in some reflection on what you've helped to bring about.

On Bluesky, doctor Atul Gawande, author of the excellent book Being Mortal (which I read in 2019) and The Checklist Manifesto (which was well-reviewed but I have not read), who was USAID Assistant Administrator for Public Health from 2022 to 2025, wrote the following posts:

I ran @USAID health programs for the last 3 years. Trump’s 90 day Stop Work Order on foreign assistance does serious damage to the world and the US. Examples:🧵

January 26, 2025 at 8:56 AM

1. Stops work battling a deadly Marburg outbreak in Tanzania and a wide outbreak of a mpox variant killing children in west Africa before it spreads further.

2. Stops monitoring of bird flu in 49 countries, a disease which already killed an American on home soil.

3. Stops critical work on polio eradication.

4. Stops >$1B in corporate drug donations and coordination eradicating tropical diseases like river blindness, elephantiasis, and others on the verge of elimination in whole regions. https://www.neglecteddiseases.gov/about/results-and-impact/

5. Stops medicines, supplies, systems building, staff support aiding >90 million women and children to get low cost vaccinations, prenatal care, safe childbirth, contraception, and other basic lifesaving health needs. https://www.usaid.gov/PreventingChildandMaternalDeaths

6. Stops direct services for 6.5 million orphans, vulnerable children, and their caregivers affected by HIV in 23 countries.

7. Stops donated drug supplies keeping 20 million people living with HIV alive.

8. Would furlough all USAID contract staff — which includes half of its global health bureau—unless exempted.

Make no mistake — these essential, lifesaving activities are being halted right now. Clinics are shuttering. Workers sent home. Partners including US small businesses face being unable to meet payroll. All despite clear requirements from Congress to do this work.

This Administration is already trashing US standing, alliances with scores of countries built over half a century, world-leading capacity and expertise, and American security.

UPDATE (5 May 2026): Gawanda estimates that the USAID shutdown has so far caused the deaths of 600,000 people, two-thirds of whom are children. Also see this ProPublica story from December 2025, "Inside the Trump Administration's Man-Made Hunger Crisis."

Books read in 2024

  Not much blogging going on here still, but here's my annual list of books read for 2024.

• James Bamford, Spy Fail: Foreign Spies, Moles, Saboteurs, and the Collapse of America's Counterintelligence (2023)
• Benjamin Breen, Tripping on Utopia: Margaret Mead, The Cold War and the Troubled Birth of Psychedelic Science
• Jennifer Burns, Milton Friedman: The Last Conservative (2023)
• Bryan Burrough, Vendetta: American Express and the Smearing of Edmond Safra (1992)
• Ron Chernow, The House of Morgan: An American Banking Dynasty and the Rise of Modern Finance (1990, 2010 foreword)
• Rich Cohen, The Fish That Ate the Whale: The Life and Times of America's Banana King (2012)
• Daniel C. Dennett, I've Been Thinking (2023)
• Cory Doctorow, The Bezzle (fiction)
• Edward Dolnick, Down the Great Unknown: John Wesley Powell's 1869 Journey of Discovery and Tragedy Through the Grand Canyon (2002)
• Jon Friedman & John Meehan, House of Cards: Inside the Troubled Empire of American Express (1992)
• Beverly Gage, G-Man: J. Edgar Hoover and the Making of the American Century (2022)
• John Ganz, When the Clock Broke: Con Men, Conspiracists, and How America Cracked Up in the Early 1990s
• Masha Gessen, The Future Is History: How Totalitarianism Reclaimed Russia (2017)
• Martin Kihn, House of Lies: How Management Consultants Steal Your Watch and Then Tell You the Time (2005)
• Stephen Kinzer, Poisoner in Chief: Sidney Gottlieb and the CIA Search for Mind Control (2020)
• Stephen Kinzer, The True Flag: Theodore Roosevelt, Mark Twain, and the Birth of American Empire (2017)
• Talia Lavin, Wild Faith: How the Christian Right is Taking Over America
• Milton Mayer, They Thought They Were Free: The Germans 1933-45 (1955)
• Michael Warren Lucas, git commit murder (2017, fiction)
• Arvind Narayanan and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference
• Craig Nelson, Thomas Paine: Enlightenment, Revolution, and the Birth of Modern Nations (2006)
• Ryan J. Reilly, Sedition Hunters: How January 6th Broke the Justice System (2023)
• Chris Rodda, Liars for Jesus: The Religious Right's Alternate Version of American History, Volume 2 (2016)
• Zoë Schiffer, Extremely Hardcore: Inside Elon Musk's Twitter
• Matt Zwolinski and John Tomasi, The Individualists: Radicals, Reactionaries, and the Struggle for the Soul of Libertarianism

Top for 2024 published in 2024: Doctorow, Breen, Ganz; other top reads for the year: Gage, Dennett, Kinzer (2020), Cohen, Gessen, Rodda

A few non-books of relevance for 2025:

What the Southern Baptists used to believe, but no longer do: https://www.sbc.net/resource-library/resolutions/resolution-on-moral-character-of-public-officials/

Umberto Eco, "Ur-Fascism," New York Review of Books, June 22, 1995

Dorothy Thompson, "Who Goes Nazi," Harper's Magazine, August 1941 (but contrast with Mayer 1955 and Gessen 2017 above)

A few planned or already (or still) in-progress reads for 2024:

G.A. Cohen, Self-Ownership, Freedom, and Equality (1995)

John Ferris, Behind the Enigma: The Authorised History of GCHQ, Britain's Secret Cyber-Intelligence Agency (2020)

Peter H. Wilson, The Holy Roman Empire: A Thousand Years of Europe's History (2017)

Lawrence Wright, The Looming Tower: Al-Qaeda and the Road to 9/11 (2006)

Arthur M. Melzer, Philosophy Between the Lines: The Lost History of Esoteric Writing (2014)

Mark S. Weiner, The Rule of the Clan: What an Ancient form of Social Organization Reveals About the Future of Individual Freedom (2013)

(Previously: 2023, 2022, 2021, 2020, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005.) 

NRA CEO Doug Hamlin's cat killing story

 Per Stephanie Kirchgaessner in The Guardian, 14 October 2024:

Douglas Hamlin, who was appointed to lead the NRA this summer in the wake of a long-running corruption scandal at the gun rights group, was involved decades ago in the sadistic killing of a fraternity house cat named BK, according to several local media reports at the time.

Hamlin pleaded no contest to a misdemeanor charge of animal cruelty brought against him and four of his fraternity brothers in 1980, when he was an undergraduate student at the University of Michigan at Ann Arbor. The charge was brought against Hamlin under a local Ann Arbor ordinance. All five members of Alpha Delta Phi were later expelled from the fraternity.

The details of the case, described in local media reports at the time, are gruesome. The house cat was captured, its paws were cut off, and was then strung up and set on fire. The killing, which occurred in December 1979, was allegedly prompted by anger that the cat was not using its litterbox.

The case caused such a furore locally that some students and animal rights activists wore buttons and armbands in memory of BK.

While The Guardian notes that Hamlin's role was not clear, Judge S.J. Elden singled him out for particular criticism as the president of the fraternity who had a responsibility to prevent it--and not, as was attempted without success, to cover it up.

(See other conservative animal abuse tagged posts, about Kevin Roberts, Bill Frist, Kristi Noem, James Dobson, Mike Huckabee, Mitt Romney, and Jerry Falwell.)